project: Performance improvements for meson build port

- Fixed CFLAGS and avoid their duplication.
- Fixed default flag application to the project on non debug builds.
- Removed unneeded meson build file from reference.
- Removed unneeded source directories from gtkdoc generation.

https://bugzilla.gnome.org/show_bug.cgi?id=783589