Kara-Moon Forum

Developers & Technology => Musical MIDI Accompaniment (MMA) => Topic started by: bvdp on November 22, 2020, 06:32:10 PM



Title: Non-standard Installs
Post by: bvdp on November 22, 2020, 06:32:10 PM
I _think_ I may have a solution for non-standard installations. A simple change to the main calling script, mma.py, now forces a check for the env variable MMA_MMADIR. So, you can set it like this:

   MMA_MMADIR=./MMA-2002 mma test

or, like any env variable in a .profile or .bashrc script.

A bit of caution:

   - we're assuming MMA-2002 has a MMA directory
   - the lib, plugins and include directories also have to be present. You can do this via links in the directory or by setting the appropriate env variables.
   - we are expanding the variable to complete pathname. I'm not sure if this is "good" or "bad".

Comments?


Title: Re: Non-standard Installs
Post by: sciurius on November 23, 2020, 10:08:51 AM
The goal of non-standard installs (funny name, since actually these should be the standard installs) is to be able to impose local material on top of the installed MMA.

This is already possible for libs, plugins and includes via the environment variables MMA_LIBPATH, MMA_PLUGPATH and MMA_INCPATH (see footnote).

For the MMA modules it would be best to rely on Python's module searching, To get the best out of both worlds I tried some experiments and got good results with this:

Evironment variable MMA_HOME can be used to designate the place where the MMA modules (and default libs, plugins and includes) are.
If environment variable is not set, the current approach of searching an appropriate dir from a predefined list of directories.

The relevant code of mma.py becomes:

Code:
MMAdir = environ.get('MMA_HOME')
if not MMAdir:
    for d in dirlist:
        moddir = os.path.join(d, 'MMA')
        if os.path.isdir(moddir):
            if not d in sys.path:
                sys.path.insert(0, d)
                MMAdir = d
                break

try:
    import MMA.main
except ModuleNotFoundError as e:
    print("Unable to find the modules needed for MMA. Please check your installation.")
    sys.exit(-1)

In my environment, I can now set up

Code:
export  MMA_LIBPATH=$HOME/lib/mma/grooves
export MMA_PLUGPATH=$HOME/lib/mma/plugins
export  MMA_INCPATH=$HOME/lib/mma
export     MMA_HOME=/my/nonstandard/location/for/mma
export   PYTHONPATH=$HOME/lib/mma:$MMA_HOME

This allows me to run mma and have it find all standard stuff at the nonstandard locaton, simultaneously being able to add/override all lib, plugins. includes and modules in my private $HOME/lib/mma library.

Does that sound sensible?

Footnote: Note that these are comma-separated, instead of the customary (semi-)colon -- it is too late to change?


Title: Re: Non-standard Installs
Post by: bvdp on November 23, 2020, 10:09:35 PM
Okay, spent some more time on this.

I've adapted the name MMA_HOME. Probably more clear.

Commas or semi-colons? Hmmm, I've always used commas. I was unable to find anything definitive on the web just now ... seems to be more an application thing than anything else. I suppose we could have a MMA_ENV_LIST_SEPARATOR variable to keep everyone happy ... but that's getting silly. Besides, in most cases I think that only one path will be used, so I'm going to leave it for now.

Note that in all the different versions setting the various LIB paths has been working just fine. Using an env variable to set MMA_LIBPATH, etc has nothing to do with the setting of MMAdir since all those pathnames are expanded to an absolute path. Side question: doesn't this really work better by setting the various paths in a rc file rather than as a env var?

What we are trying to do here in mma.py is to make robust system for finding the main mma tree, no matter how convoluted the installation is.

So, here's the new "logic":

   - if no env variable found we create a path list consisting of some system-specific locations (have a look at the source) depending on the system,
   - the system path (where python expects stuff) is appended to the list created above. So, if MMA is not found in one of the system-specific locations we default out to the system install.

   - finally, if (and only if) a env variable MMA_HOME has been set we wipe out all the stuff above and look only in the passed location. If you have specified a location for MMA and it doesn't exist or is not reachable you WANT an error ... you don't want to blindly continue in an unknown location.

Oh, thanks for reminding me that except/pass is my friend :)
 
New mma.py attached.


Title: Re: Non-standard Installs
Post by: sciurius on November 24, 2020, 11:14:57 AM
Close, but no cigar  :)

By explicitly prepending the MMAdir to sys.path you defeat the setting of the PYTHONPATH environment variable. That's why in my proposal I do not touch sys.path if MMA_HOME is set.

Also:
Code:
        try:
            import MMA.main
        except:
            print("Found MMA directory in '%s'. Unable to execute MMA.main.py." % MMAdir)
        sys.exit(-1)

This will always terminate with sys.exit(-1), I'm quite sure that is not what you intended.

Quote
Oh, thanks for reminding me that except/pass is my friend
You're welcome... But remember you already found out (the hard way) that you should never catch arbitrary exceptions...


Title: Re: Non-standard Installs
Post by: bvdp on November 25, 2020, 02:10:54 AM
Close, but no cigar  :)

By explicitly prepending the MMAdir to sys.path you defeat the setting of the PYTHONPATH environment variable. That's why in my proposal I do not touch sys.path if MMA_HOME is set.

Olay, let me digest that. Not sure how the python running mma finds anything this way. But, it does make sense. Let me look tommorrow.

Quote
Also:
Code:
        try:
            import MMA.main
        except:
            print("Found MMA directory in '%s'. Unable to execute MMA.main.py." % MMAdir)
        sys.exit(-1)

This will always terminate with sys.exit(-1), I'm quite sure that is not what you intended.

I think this is almost correct ... however I  do need to change the except to

    except ImportError:

So that if there is an error in the script it doesn't think it should fall though to the bottom.


Title: Re: Non-standard Installs
Post by: sciurius on November 25, 2020, 07:24:34 AM
Quote
I think this is almost correct ... however I  do need to change the except to

    except ImportError:

So that if there is an error in the script it doesn't think it should fall though to the bottom.

Yes, but more important is to shift the sys.exit(-1) to the right so it becomes part of the except handling. As it is now when MMA.main completes normally, it will always sys.exit(-1) and I don't think that is desired.


Title: Re: Non-standard Installs
Post by: sciurius on November 25, 2020, 12:27:07 PM
There's another python surprise getting in the (well, my) way...

Whatever value you set PYTHONPATH to, python always prepends the script directory. I have to rethink....


Title: Re: Non-standard Installs
Post by: bvdp on November 25, 2020, 06:46:47 PM
Here's yet another version of mma.py.

I have fixed the sys.exit() issue by creating 2 of them. One for success and another for failure :) And, don't ask where I got the -1 value from ... some kind of mind-fart no doubt.

As far as PYTHONPATH goes ... that's got nothing to do with MMA. Python will process that before running mma.py. I've never had occasion to use that myself. I read about a number of people using virtual environment for python and that might be a better way to run?

It is doing better failure reporting if MMA_HOME doesn't work out as well.


Title: Re: Non-standard Installs
Post by: bvdp on November 25, 2020, 09:37:13 PM
More on the path separator ... in python there is a constant os.pathsep. In unix (linux and I'm assuming mac) it is set to ":" and in windows ";".

We use comma separated lists right now in MMA when using spaces doesn't work. Which is, probably, why I used a ',' in the env stuff. However, simple enough to change that for the env stuff like LIB_PATH, MMA_HOME, etc.

Comment?


Title: Re: Non-standard Installs
Post by: sciurius on November 26, 2020, 09:43:53 AM
in python there is a constant os.pathsep.
Good!

Quote
We use comma separated lists right now in MMA when using spaces doesn't work. Which is, probably, why I used a ',' in the env stuff. However, simple enough to change that for the env stuff like LIB_PATH, MMA_HOME, etc.
It sounds definitely like a good idea to use os.pathsep in the environment variables. It makes setting these consistent with the rest of the system.

From the perspective of my workflows I do not see a real use for SetLibPath and friends now we have the environment variables. MMA should be about making music, not system path names :).


Title: Re: Non-standard Installs
Post by: sciurius on November 26, 2020, 01:12:08 PM
I went back to the drawingboard one more time and my current opinion is that there is only one 'right' approach: make MMA installable using pip.
See e.g. https://packaging.python.org/tutorials/packaging-projects/ .

This involves a bit more than tweaking some paths but should make install and run sufficiently robust. A pip-style source distribution is also ready to be run directly from source.


Title: Re: Non-standard Installs
Post by: sciurius on December 04, 2020, 10:21:33 PM
Warning -- Work In Progress!

Attached are two scripts.

setup.py is a standard python installer script, to be used with the 'pip' tool.

To use it, a few mods to the mma sources are needed: Directories egs includes lib plugins util must be moved to below the MMA directory, and
a new directory cli must be created and the attached 'mma' script must be put there.

In commands:

Code:
tar xf mma-devl.20.02f.tar.gz
cd mma-bin-20.02f
mv egs includes lib plugins util MMA
cp ... setup.py
mkdir cli
cp ... cli/mma

For a user mode install, type
Code:
pip install --user .

For a system install, type
Code:
sudo pip install .

This should work on Linux, MacOs and Windows.

Thoughts?


Title: Re: Non-standard Installs
Post by: bvdp on December 04, 2020, 11:23:08 PM
Without running code ... does this mean that the modules like chords.py, main.py, etc are sharing the same dir as lib, plugins, etc? That doesn't excite me.

I assume that the calling program, mma.py, will live outside of this directory? Or will it work there or in a bin directory somewhere else?

One thing that has always bothered me (not a big bother, but it's a bit ugly) is my decision many years ago to use MMA for the modules ... at the time I hadn't thought thorough the case issues this could present in windows/mac. I guess it works okay, but I'm wondering if that should be looked at?

I am planning to release the current stuff as 20.12 in the next week. So, we should look at packaging for 2021?


Title: Re: Non-standard Installs
Post by: sciurius on December 05, 2020, 09:42:02 AM
This is what I get:
Code:
/usr/lib/python3.7/site-packages/MMA/after.py
/usr/lib/python3.7/site-packages/MMA/alloc.py
...
/usr/lib/python3.7/site-packages/MMA/includes/...
...
/usr/lib/python3.7/site-packages/MMA/lib/...
...

This is a convention used by most python packages.

I'm going to move the egs and util directories elsewhere (/usr/share/mma/egs and /usr/share/mma/util) since their contents doesn't need to be installed next to the mma code.

The cli/mma program is a new script that will be the installed as '/usr/bin/mma'. The current 'mma.py' script will continue to work in an uninstalled mma file tree.

This is a work in progress so no reason to hurry it in 20.12.

Nice things I found so far: It is trivial to perform a local install (pip install --user) and test everything. pip uninstall will nicely clean up everything.

Imagine all users can install/upgrade mma by just typing 'pip install mma'... No downloads, unpacks, ...


Title: Re: Non-standard Installs
Post by: sciurius on December 05, 2020, 02:49:33 PM
Hmm. It seems that Hugo Neves de Carvalho <hugonvsc@gmail.com> already provides a good working pip-installable MMA. Maybe it is time to join efforts?

Code:
Metadata-Version: 2.1
Name: mma
Version: 1.5.20.2
Summary: MMA - Music Midi Accompaniment available as a pip module
Home-page: https://github.com/hugonxc/mma_pip
Author: Hugo Neves de Carvalho
Author-email: hugonvsc@gmail.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown

# MMA—Musical MIDI Accompaniment

This is a fork from [MMA](https://www.mellowood.ca/mma/) to make it available as a pip module.
The current version of MMA in this project is 20.02, which can be different from the main repository.


Title: Re: Non-standard Installs
Post by: sciurius on December 05, 2020, 03:30:03 PM
Updated setup.py. This one copies the contents of egs and util to /usr/share/mma (or ~/.local/share/mma).
The directories egs and util must not be moved to MMA.


Title: Re: Non-standard Installs
Post by: bvdp on December 07, 2020, 04:56:35 PM
I've been thinking about this install business and, at this point, I'd like to maintain the current scheme of having a tarball and deb on my site. But, there are real good reasons for having alternates like RPM and PIP.


Title: Re: Non-standard Installs
Post by: sciurius on December 08, 2020, 11:31:00 AM
No problem. I'll take care of the RPM.


Title: Re: Non-standard Installs
Post by: sciurius on December 13, 2020, 01:56:51 PM
Some setup/distribution tools are not happy with the format of the devel version number. 20.02 was okay, but I got some complaints about 20.02f. It is not a big deal, but if it doesn't really matter to you it would be nice if the next devel version was 20.12.1 instead of 20.12a.


Title: Re: Non-standard Installs
Post by: bvdp on December 13, 2020, 04:34:53 PM
No problem. Consider it done.