Kara-Moon Forum

Any thoughts on converting mma to python 3.x? And, no, it's not just a matter of running 2to3!

I have started the project and it's not going to be trivial. A lot of things (like converting print to print() ) are trivial. And converting print output to display properly is just a PIA.

A big chuck of the time will be in converting the way internal representations of MIDI data are stored and manipulated. Since python 2.x did not have a true byte array the individual datums in MIDI data were all stored as text characters. So, to generate a 2 "character" integer we did stuff like:

     v = chr(int(x) >> 8 & 0xff) + chr(int(x) & 0xff)

which did 2 expensive (and, frankly, unintuitive) things:

     1. String concatenation ... lots of cpu time
     2. conversion to chr() ...

In python3 there is a concept of bytes. So, the above becomes:

     v = bytes([int(x) >> 8 & 0xff, int(x) & 0xff])

which generates the same 2 character array. But, it's now in bytes, not a string.

I think that just this should makes things faster. And, even if it doesn't, it'll be a lot clearer!

So, my plan at this time is to slowly convert the code to 3. Won't be done tomorrow!

But, at the same time, I'd like to modify some things in the current program to make it all nice and consistent. One example, the TEMPO command permits things like:

      Tempo +20 5.5

which increases the tempo by 20 bpm over 5.5 bars. I'd think that something like:

      Tempo +20 bars=5.5

might be more readable and in tune with other options in the program.

I have a feeling that 13.12 (the current version) will be left for use by folks with old python.

What do you guys think about all this?

     

Surely the answer would be to use lists. That would then work for both. I believe each MIDI command has a maximum of 3 bytes so your basic list would be:
command = [0,0,0]
You would then make a list of lists for however many commands you want to keep current.

c_list = []
for i in range(number):
  c_list.extend([command])

you now have a 2 dimensional addressable array in the form:

value = c_list[n1][n2]
c_list[n1][n2] = value


OK, these will actually be integers, but I've used something like this for byte values with RS232 serial stuff without any problems.


you can also do whole rows:

c_list[n1] = [1,2,3]
v1,v2,v3 = c_list[n1]

... or am I missing something ::)

Well, yes ... and well, no :)

midi commands can be just about any length. A byte in the preamble signals the length.

I use a lot of lists and dicts in mma. The internal midi is actually a dict of lists.

The problems arise in the funny way values are represented in midi. Long story, but has to do with the good old days when we didn't have buckets of memory. Short story, they are a complete PIA.

So, at some point one has to convert values into 2 and 3 byte STRINGS which are then included in the midi.

Having a single version of mma for python 2 and 3 was considered. I just don't think it's workable. But, we'll have to see.

When I started writing mma 9 years ago python didn't have a real byte type. So, it was all faked in strings. And, a complete pain that has been. Now, with 3 having real byte (arrays) and a lot of other nicites it'd be a real shame not to use them :)

So far, it looks like my 3 version MIGHT be back-portable to 2. Don't really know at this point.

Seriously, would python3 only be a game stopper for anyone? I think all the linux distros are shipping with both, some with only 3. Windows needs to dl anyway. I do think that, eventually, 3 will the standard. Just don't know when eventually is :)

Damn, I get all excited about python 3 and then I read this ...

  http://alexgaynor.net/2013/dec/30/about-python-3/

Scary thing is, I think he's probably right.

Bit of a chicken and egg situation really :(

Damn all the chickens and eggs!

Seriously, after making this huge announcement ... I'm having 2nd thoughts. What I think will happen is that I'll work on isolating stuff which will effect the 2 to 3 change into functions which might just work with some conditionals. Print() is a biggie, but there is no problem to have all the prints in the existing program change to something like my_print(). My string/midi packing stuff is being converted right now to using the struct.pack() command in python.

So, the next version of MMA will most probably be python 2. Depending on how hard the changes are, it might be 2/3. All I can say is stand by!

I agree print is a bit of a boogar. I think it was V2.6 onward that accepted both forms, but have no idea how much usage there is of older versions.

One possible solution is to use sys.stdout.write({string})

It's a bit more work, but encapsulated in your own function(s) to make sure you're converting all numbers to strings, it should work equally across all versions.

Just to keep the curious up-to-date, I've been working on making mma work on python 2 and 3.

At this point I have restructured the code which creates the midi data strings from using chr() to a real byte type. This should work on both 2 & 3. I have to check to make sure that no other manipulators are changing things, but I think it's pretty solid.

I have changed all the print statements to print(). For 3 this will be a function; for 2 this simply prints the contents of a solitary tuple. There will be some minor differences in the way LFs are handled, but I don't think anything important will be lost.

The import functions have been changed to work with 2 and 3.

Classes containers have been modifed.

So, we're making progress.

Next up are checking integer division. A biggie :) In 2 the result of 1/2 is 0; in 3 it is .5. If one wants 0 you have to use //.  This works in 2 and 3 so not a big deal.

After all that, there will be some need for conditional code (or the use of an exception) to mangle things into both pythons. But, I'm hoping that this will be painless.

BTW, with the stuff I've done so far I've found a number of minor bugs (fixed!) and managed to make the whole mess run about 1% faster. I'll be putting up a developer version in the next week or so. Hopefully you guys can find some more bugs for me :)

Glad to see you found a silver{ish} lining ;)

I'd quite forgotten about the oddities with division. The one that caught me out for a while was getting division inside functions to produce floating point values when the function's input values happened to be integers.

I tried to run MMA from python3 yesterday. Still lots of things to work on. Biggest problem, for me, is the difference between byte and old-style character strings. I'm using struct.pack() to create the MIDI data and there are some oddities to overcome (that is a nice way of saying it).

Oh well, no rush. Neither python2 or python3 are going away anytime soon :)

Well, congratulations to me  :-

I just complied a successful midi file using the python2/3 version of MMA.

Yup, mma is working with python3 :D

Now, I don't know yet if any errors have crept in, and I'm sure they have. But, the midi file is playable and sounds the same as using python2.

This is huge progress and I'm pretty excited (I'm not so sure I really appreciate all the constraints a typeless language like python is putting me though to force typeness ... had I know this I might have just used C in the first place).

(http://www.clicksmilies.com/s1106/spezial/jasons_smilie/juggle.gif) :-  ;D

Don't get the party hats and noise makers out just yet. The python3 version only works on about 80% of the song files I have. The rest dump core ... not nice. But, the errors appear to be mostly integer division stuff. Be awhile before a developer version goes up here.

But, progress is progress!

I have now successfully run MMA under Python 3.3 on all 1000+ files on my local computer. This includes all the example files in the distro. I've not listened to each and every file since this would be about 50 hours worth of listening ... and I do have snow to shovel!

So, it appears to be working :)

Only downside is that python 3 seems to run 10 to 20% slower than python 2. Not a big deal, but interesting.

I need to fix a few tests in the main mma.py program which tests python versions and some other little stuff. I'll put up a developer test version later this week.

I have just uploaded mma-13a, a test version which supports Python 2.5+ and 3.x.

The calling script, mma.py, has a first line:

     #!/usr/bin/env python

To force the use of python 3.x you can do several things:

   1. Call the program using the command line "python3 mma.py"
   2. Change the first line from "python" to "python3"
   3. Check the file "python" on your computer. In most cases it is a sym-link to
       either /usr/bin/python2 or /usr/bin/python3. Modify as necessary

This version of mma has a few conditional switches to use 3 or 2 code (and a few is actually 2).

Would you guys please bang on this and see what I've missed.

   http://mellowood.ca/mma/downloads.html#developer

Thanks!

Put a new version this morning to fix 2 small bugs in the -G code.

Keep those bug reports coming!!

Yet another update. I've fixed some little things in mma itself and converted the major utilities.

This seems to be coming together very nicely. Sure hope it is of benefit.

Very pleased it's coming on, but afraid I can't do any bug-hunting at the moment. I'm more-or-less full time on squashing Yoshimi bugs, having got myself on the dev list{me and my big mouth}  ::)

Bug hunting is so much fun. If I'd had any idea of how large a project this might be ... I'd be a lot smarter than I am :) But, seriously, I would have written things a bit differently and included testing stubs in each section. I'm not sure that tests would cover every eventuality, but they might have made finding some things easy. On the other hand, who knows for sure, Much bigger projects than MMA have no testing base and are more obfuscated :) And, for the most part, seem to work.

Another new version for ya all. Fixed an silly bug in the volume routines (gotta be very careful in python when using integer vrs fp division). Also rewrote the midi input part of MidiInc into it's own class (with the idea it might be callable for other things).

BTW, I'm now using this as my production version. Generates midi files just great.

Have fun.

I've played a bit with this dev version.
Works well with Python 3.
No issue seen so far.
Great job :)

Alexis

Thanks. Do let me know if you find something.

I have some more silly tests to run from my end. Want to make sure the various command options are doing what they are supposed to do.

Just a little heads up ... using python3 the midisplit command barfs as well as the -b/B command line option. I know where the problem is, and what it is, but have no idea just now of how to properly fix it for both py2 and 3. I'm giving it a day to percolate though my grey cells.
 

So, got the latest set of bugs quashed. This insistence by python3 in keeping different types of data (ie, integer, string, bytes) all different really is a pain. Mind you, once it's programmed it is nice and forces things to be "right".

One of the nice things about doing this is that you can stick a bunch of 1 byte values into a string, convert it to a bytearray and then, instead of calling up ord() and chr() all the time to read individual bytes in the array, you just examine then with standard string splicing notation.

Mind you, if you do it in one spot ... you really need to do it everywhere ... otherwise the whole tower collapses and you wonder just what in *#(*$ is going on.

I'll try to post a new version to bang on tomorrow. It seems to be PERFECT here :)

Yet another test version (13.12e) has been uploaded.

Fixed a number of little things with the 2/3 code. But, also managed to fix or cleanup some outstanding bugs and issues with the base code.

This is getting close to a real release :)

I was playing my sax last night with, of course, MMA generated tracks. Oh, my ... one song had the most awful sounding part. Oppps .... used to work. Anyway, I buggered the fermata code when doing the 2 to 3 conversion. All fixed now, and a upload of test version "f".

Last one for now?