New process to build MacOS version
ddeanbrown Implementing something like that to autotools based build would probably take a lot of effort and time, so maybe not worth it. The implementation/testing "time" part means also substantial part of the remaining "autotools used in freeciv build" -time getting used to it - less time for the system to be in production use. Better to target meson based build directly instead?
That would mean:
- New system would not get introduced to S3_0 (freeciv-3.0) as there's no meson support there
- While meson based build is likely to remain "experimental" in S3_1, I see no problem in using it in a statically scripted build (there's not going to be surprise changes in the environment, to which the build system would need to adapt)
- First step is to get meson based build to work (maybe it does already?) and tested on MacOS. That's something to do in any case, and I'm soon starting to look at it
ddeanbrown Agree about going to meson directly, since it looks like meson is the way things are going already and meson claims to support MacOS.
On CI, Meson based macOS (homebrew) build has now been working (build going through) for 2.5 months. Could someone test if a macOS build made with Meson actually works? There's not much point to start working on packaging (implementation) of something that doesn't even work.
ddeanbrown I can test it, just need to know how to get it onto my Mac. Maybe a link to where it lives on the CI? or attach it to this ticket?
ddeanbrown)
Reply To ddeanbrown
I can test it, just need to know how to get it onto my Mac. Maybe a link to where it lives on the CI? or attach it to this ticket?
Currently CI just does build tests, there's no artifacts stored anywhere. And as there's no packaging yet, there even wouldn't be a single artifact to send anywhere. So you would need to build it yourself.
ddeanbrown I can deal with the packaging, but don't know how to build Meson based executables myself.
I don't understand how the CI works - is it just checking if dependencies are satisfied? Or does it build executables and then just doesn't save them? I can imagine having dependencies be satisfied and building still possibly failing.
Reply To ddeanbrown
I can deal with the packaging, but don't know how to build Meson based executables myself.
It shouldn't be that different from building with autotools. It has the same steps, instead of ( configure + make (+ make install)) it's done with ( meson + ninja (+ninja install))
doc/INSTALL.meson has the documentation.
Example of building S3_1 (that still requires you to ack that you're using experimental build system), assuming freeciv sources are in subdirectory 'freeciv-3.1' under current directory:
mkdir build
cd build
meson ../freeciv-3.1 -Dack_experimental=true
ninja
There's script 'run.sh' produced to run freeciv.in the build directory (i.e., to be used like ./fcser, ./fcgui, etc in autotools builds):
./run.sh freeciv-gtk3.22
The build process is currently tested (in CI) for gtk3.22-client only. That's being extended in #45929, and an issue has already been detected with Qt-client build.
ddeanbrown ddeanbrown Thanks to your comment on 10-17 alerting me to the existence of doc/INSTALL_meson I got inspired to try building. The good news is I succeeded in building a freeciv-gtk3.22 executable that works. The bad news is that there are still some problems. Details are lengthy, so I put them in a text file "MacMeson.txt", attached to this ticket. I had to hack the meson.build file, patch file is also attached. Since I'm a meson newbie, I expect there may be better ways to do what I did. I tried to not break it for non-Mac builds, but that should also be checked.
Thanks. Need to compare your results with the CI build logs to see what makes the difference that it builds successfully there (except Qt side - test builds ongoing to fix that). Maybe some packages that it has installed (See 'mac-meson' part of .github/workflows/ci.yml ), or environment variables & configure parameters ( 'mac-meson' case in scripts/ci-build.sh ):
As for the installed program not being runnable, did you test just the client, or also server alone? I can't easily test client on CI, but the test to run server (and autogame on it) from #45997 works there (just be aware of #46010 with the very current freeciv repository version)
In general I think that best RoadMap would be to first try to replicate the exact same process that CI uses to create a successful build. If you get that working, we could then start fixing meson build issues with other kind of workflows.
As for the failures to find correct dependencies; do you have pkg-config or pkgconf, or neither installed? I think either one should work fine, but if you have neither, it will force meson to use some fallback mechanisms to search for those components, and I wouldn't expect those mechanisms to work too well with this setup.
Also, I don't know how much spare time you have to throw to this, but it might also be worth making an autotools based build from the same freeciv-3.1 sources, to see which of the problems are specific to meson based builds, and which ones are present regardless of the build system.
ddeanbrown Finally got a chance to dig some more, and made good progress. Learning of the existence of .github/workflows/ci.yml and scripts/ci-build.sh helped a lot. Also figured out that having done an install of MacPorts caused my $PATH setting to change such that the MacPorts version of pkg-config was getting found. Fixed that and now meson reports - "Found pkg-config: /usr/local/bin/pkg-config", which is the Homebrew version. With a fresh pull from S3_1 I was able to build clients gtk3.22,sdl2,gtk4 and fcmps gtk3,gtk4,cli and freeciv-server - all of which launch OK and run (maybe with bugs). I mostly followed the steps in the "mac-meson" section of scripts/ci-build.sh, the main exceptions are the line -
export PKG_CONFIG_PATH="$(brew --prefix icu4c)/lib/pkgconfig"
was not needed, and the CPPFLAGS line needs an addition thus -
export CPPFLAGS="-I$(brew --prefix readline)/include -I$(brew --prefix icu4c)/include"
which partially matches what is in the "os_x" section. Also the -Dsyslua=true fails. Just one change in meson.build - the icu_dep, patch attached. The problem I had before with @rpath vanished, and I made a working app bundle. I still need to try qtver=qt6, and compare an autotools based build.
ddeanbrown ddeanbrown ddeanbrown Fixed the -Dsyslua=true problem by doing "brew install lua". Fixed the qt6 problem by doing "brew install qt6". Now have successfully built and run clients gtk3.22,sdl2,gtk4 and fcmps gtk3,gtk4,cli and freeciv-server with meson command line that more closely matches what is in "mac-meson" section of scripts/ci-build.sh.
Also got newer version of meson: 0.64.0, which gives this warning -
WARNING: Running the setup command as meson options instead of meson setup options is ambiguous and deprecated.
So my meson command line now looks like -
meson setup /Volumes/Hogwarts2/freeciv/git/freecivS3_1 -Dack_experimental=true -Dclients=gtk3.22,sdl2,gtk4 -Dfcmp=gtk3,gtk4,cli -Druledit=false -Dsyslua=true -Ddebug=true
ddeanbrown ddeanbrown Excellent progress - I can now build an app bundle by running scripts/ci-build.sh. Resolved the issue with the icu_dep in meson.build, the export PKG_CONFIG_PATH="$(brew --prefix icu4c)/lib/pkgconfig" is needed and nothing needs changing in meson.build and I deleted that patch. New patch for scripts/ci-build.sh with a new section for "mac-meson-appBundle".
Issues:
1 sometimes running the scripts/ci-build.sh with target "mac-meson" or "mac-meson-appBundle" produces clients which fail to run, they get error msg -
dyld: Library not loaded: @rpath/libfreeciv.dylib
zsh: abort /Users/Dean/freeciv/mac-meson/bin/freeciv-gtk3.22
I still don't understand this @rpath stuff. The modpack installers and server run OK.
2 when I did get the "mac-meson-appBundle" target client to run, it fails to find the ...share/freeciv directory. I investigated the chain starting in meson.build has get_option('datadir') and priv_conf_data.set('DEFAULT_DATA_PATH'... , then fc_config.h has
#define DEFAULT_DATA_PATH ".:data:/3.1:/Applications/Freeciv.app/Contents/share/freeciv" (which I'm not sure is correct), and then shared.c has
data_dir_names = base_get_dirs(NULL != path ? path : DEFAULT_DATA_PATH);
but I don't understand it well enough to know if there's something not working right. But seeing
path = getenv(FREECIV_DATA_PATH)
made me realize I do a work around by defining an environment variable value for FREECIV_DATA_PATH at run time, which is easy to do in the iInfo.plist file in the app bundle. So that's what I did, also needed one for FREECIV_SCENARIO_PATH, and it all works.
ddeanbrown ddeanbrown ddeanbrown ddeanbrown Researched @rpath and dylib's and learned that searching for dylib's at run-time can be configured with environment variable LD_LIBRARY_PATH. Easy to add that to the Info.plist file in the app bundle, and now everything works. Updated the ci-build.sh.diff patch.
I also learned that when LD_LIBRARY_PATH is not set, /usr/local/lib still gets searched. And when no -Dprefix="..." is passed to meson the install defaults into /usr/local, with the executables in /usr/local/bin, which is part of the default PATH setting so running the clients from anywhere works. So I took the liberty of changing the "mac-meson" section of scripts/ci-build.sh to do that. If you're not OK with that you can leave that section as is.
With this patch I think our work here is done and this ticket can be closed.
ddeanbrown ddeanbrown ddeanbrown I was too hasty - further testing found a couple of bugs which I have fixed with a newer ci-build.sh.diff patch. This diff is based off Nov 21, 2022 version of ci-build.sh.
Which now builds a freeciv-qt client that runs OK.
I also found another bug - I will make a new ticket https://osdn.net/projects/freeciv/ticket/46134. The meson built clients fail to find downloaded rulesets in ~/.freeciv/3.1 BUT clients built with autotools are OK.
ddeanbrown)
It's good to have the working script fragment, but we need to move it somewhere else from ci-build.sh. You're not really doing CI like the name of the script tells (it's run from github actions, as instructed by .github/workflows/ci.yml. This is not just a cosmetic thing, but means also that the script is not included in the release tarballs.
I hope to get this done before alpha5 so that release kind of build could be tested with it already, but latest with beta1.
ddeanbrown With the fix from the patch in https://osdn.net/projects/freeciv/ticket/46134, can simplify ci-build.sh, so replacing the ci-build.sh.patch file.
I've been running ci-build.sh from my command line, and as long as I can get it from the git repo it doesn't matter to me if it's in the release tarballs. I'm also OK with putting my new stuff somewhere else.
If the end goal is to include a script in the release tarball that Mac users can run to produce an app bundle, then I think would also want a README.MacInstall file to explain how to do it. I can also continue to build new app bundles myself and put them on dropbox.
ddeanbrown Reply To ddeanbrown
With the fix from the patch in https://osdn.net/projects/freeciv/ticket/46134, can simplify ci-build.sh, so replacing the ci-build.sh.patch file. I've been running ci-build.sh from my command line, and as long as I can get it from the git repo it doesn't matter to me if it's in the release tarballs.
Release builds should preferably be built from the tarball, to make sure that it match exactly what's in the tarball. In any case you can't distribute a source tarball and a binary that cannot be built from those sources.
If the end goal is to include a script in the release tarball that Mac users can run to produce an app bundle, then I think would also want a README.MacInstall file to explain how to do it.
That's an obvious goal, as it's required for GPL compliance.
Attached patch puts your script fragment to a new script under platforms/
- Some error checking added
- Removed special log handling after meson failure that only makes sense when running inside github actions (and this script does not)
- Removed "-Ddebug=true" if this is meant mainly for release builds
- Master only: Removed "-Dack_experimental=true"
Can you test if this works like this?
ddeanbrown I had just been thinking I should move my shell script stuff to a separate file, thanks. This does work OK.
There's a couple more tweaks I can think of, but this is good for now.
Reply To ddeanbrown
I think would also want a README.MacInstall file to explain how to do it.
Please open a new ticket about this.
I can also continue to build new app bundles myself and put them on dropbox.
I'd prefer getting things set up so that those would be put under freeciv's usual (official) downloads locations. That could be a good goal for freeciv-3.1, wouldn't it? So first such build would be 3.1.0-beta1, and preferably most parts of the process tested already with 3.1.0-alpha5.
ddeanbrown New ticket created for README.MacInstall - https://osdn.net/projects/freeciv/ticket/46262
Maybe the homebrew-appbundle.sh script should also install Homebrew and the needed Homebrew packages, or there could be another script to do that. Maybe open another ticket for that?
Clone the build process used by Longturn fork to produce MacOS app bundle. Discussed on the freeciv-dev mailing list. Some relevant bits -
Dean wrote:
"Their resulting Mac app bundle does not contain any dylibs and is completely self-contained and path-independent, so for example it can be renamed or moved to another directory. That's a big improvement. It's also much smaller, 233 MB vs 891 MB."
Louis wrote:
"Our Mac executable is created on Github Actions. We fetch build tools (cmake etc) from Homebrew and then rely on vcpkg to get a static build of Qt and the other dependencies. The code is there:
https://github.com/longturn/freeciv21/blob/master/.github/workflows/build.yaml#L173-L218"
"I think Freeciv could clone what we do, though it would be a bit harder for a project using autotools. vcpkg is a package manager by Microsoft that builds everything from source, so just installing Qt takes about 3 hours. Adding several Gtk and SDL versions would make this even longer (though Github Actions can cache the results if they're not too big). Marko has a working build script on Github Actions 1, maybe you can start from there and just add the packaging step..."
"vcpkg is very well integrated with CMake: it replaces find_package to automatically build all the dependencies. I think it's simply impossible to do in autotools, so one would need to install packages manually via the vcpkg CLI and then pass all the paths to configure. To make things even harder, vcpkg encourages static builds."
Dean wrote:
"I thought that vcpkg doing static builds was a good thing for what we want."
Marko wrote:
"Well, if it works with direct CMake use, one would expect it quite easy to get it to work with Meson set to use CMake as the dependency find method."
"Be careful when working with any freeciv21 code to not infringe their GPLv3+ license. When you submit anything to freeciv, you're making it available under GPLv2 too (freeciv is GPLv2+ licensed)"
This would supersede #44400 & #44399.