Also, when did I start getting to a point where I could legit complain about Python OOP features?
Thursday, February 23, 2012
Python class constructor polymorphism
C++ has this neat class functionality where it will choose the correct class constructor based off what arguments are passed during initialization of the class. Python seems to lack that. I want to be able to pass either a string to the class constructor method OR a reference to an already opened file. It would make my class so much more functional. It seems I'm forced to make one class for each scenario. Well, maybe I could do some type detection and make some decisions in the class constructor based of what is passed, but that's.... hacky?
Good blog post on *args sand **kwargs
http://www.saltycrane.com/blog/2008/01/how-to-use-args-and-kwargs-in-python/
I haven't found myself needing variable argument lists in functions, but I see it frequently in other people's code. Variable function argument lengths in C is something I understand on a superficial level but have never needed to use myself - but I almost never see it in other code I read.
I haven't found myself needing variable argument lists in functions, but I see it frequently in other people's code. Variable function argument lengths in C is something I understand on a superficial level but have never needed to use myself - but I almost never see it in other code I read.
Monday, February 20, 2012
Document
I blocked off some time at work to knock out some more of the tool I'm building for extracting/analyzing support cases. It's been about a week since I had last dug into it so it was tough to figure out where I had left off. Since I didn't have much time before I had to be home I decided to abandon coding and start documenting. I wrote out the general reason for the program, some sample usage of the main class and its instance variables and methods, and documented the purpose of the methods I had already written.
Honestly it was a weird form of procrastination, but now I know exactly what's done and this is a good pointer for where I need to go. There was a link on Reddit I wish I had bookmarked about programming "backwards" where you write the documentation first - even writing out sample implementations of APIs not written yet. It provides a sort of design document and gives direction of what the actual implementations need to do.
I'll probably write about this more later (actually maybe I already wrote about it...) in a specific sense. The largest struggle I've had with this program is how to store results. Should the class members return data internally, or to an outside variable? My experience with C has driven me to the latter. Most functions return the value you're asking it to compute. Sometimes the function will work with pointers to outside variables and technically return void (or 1 or 0 or whatever means success) but it still involves outside variables. Once you start working with classes you have to somewhat abandon those rules and try to stay within the class. I am always repeating this: "A class is where you keep data and the functions that can alter that data". A natural extension of that is "data in the class that you modify will also be in that class".
I struggle with object oriented programming. This must be something a CS major puts a great deal of time into, and maybe that's why Java is a dominant language in CS programs.
Honestly it was a weird form of procrastination, but now I know exactly what's done and this is a good pointer for where I need to go. There was a link on Reddit I wish I had bookmarked about programming "backwards" where you write the documentation first - even writing out sample implementations of APIs not written yet. It provides a sort of design document and gives direction of what the actual implementations need to do.
I'll probably write about this more later (actually maybe I already wrote about it...) in a specific sense. The largest struggle I've had with this program is how to store results. Should the class members return data internally, or to an outside variable? My experience with C has driven me to the latter. Most functions return the value you're asking it to compute. Sometimes the function will work with pointers to outside variables and technically return void (or 1 or 0 or whatever means success) but it still involves outside variables. Once you start working with classes you have to somewhat abandon those rules and try to stay within the class. I am always repeating this: "A class is where you keep data and the functions that can alter that data". A natural extension of that is "data in the class that you modify will also be in that class".
I struggle with object oriented programming. This must be something a CS major puts a great deal of time into, and maybe that's why Java is a dominant language in CS programs.
Thursday, February 16, 2012
Reading bytes in Python
Today's lesson in why you should RTFM, but more on that later.
One of the biggest successes I had while learning C was reading text and binary data from files. I'm going to tackle the same task in Python. The first step is going to be reading in some data (the first 136 bytes) of the file header.
The file I'm reading is the same sort of file I learned to read in C - a file with some header data with various things like the date it was acquired and some comments, and then large chunks of digitized analog data (like a .wav file in a way).
I'm starting small - I only want to read in the first 136 bytes of the file. The first 4 bytes represent an integer that is always in the file (sort of a marker that tells us where it came from). The next 4 bytes is the version of the file format (also an integer), and the next 128 bytes represent a string of 128 characters (which are 1 byte each, so 128 characters).
I've spent a good deal of time prepping for this task - most of the info I needed was in the documentation for file objects and the struct module. In short, I'm going to read in a known number of bytes using the read() method for file objects, and then "unpack" those bytes into a specific format (integers and character strings) using the unpack() function in the struct module. So, here's a start:
I once read somewhere that using a class full of empty instance (self) variables is a good way to mimic how structures in C look. I don't know if that's very "Pythonic" but it works for me. In the PHeader class I've defined three variables that I'm going to fill in with data from the file. The actual code that executes is under "if __name__ == '__main__'", which is just a fancy Python way of saying "if this .py file is run on its own then do what's underneath".
First I open the file as "p", and initialize "s" as an instance of PHeader. I know that p.read(N) will read in N bytes of the file, so I need to somehow tell Python to interpret those four bytes as an integer (as opposed to another data type that is 4 bytes) and then make s.MagicNumber equal that resulting number.
So that's where the struct module's unpack() function comes in. unpack() has this prototype: unpack(fmt, string). fmt is the format of the bytes being read (we want an integer so we pass it "i") and string is the bytes to unpack. Well, the result of p.read(4) is our string, so this line...
s.MagicNumber = unpack('i', (p.read(4)))
...gets our four bytes, interprets that as an integer, and passes the result to s.MagicNumber. A big caveat that I missed while writing this that caused a great deal of confusing is that it doesn't ACTUALLY pass JUST the integer. It passes a Python data type called a tuple with the integer I wanted as the first element of that tuple. Tuples (and other Python data types) work a lot like arrays in other languages - but more on that in a second.
The next line does pretty much the same thing...
s.Version = unpack('i', (p.read(4)))
Ok cool, we have now read two integers from the file and stored them in some variables. This next part is tricky and caused a lot of wailing and gnashing of teeth on my part. The struct documentation told me that "i" is the format character for integers, and "s" is the format character for character arrays (strings). Since the next 128 bytes of the file is a row of 128 characters (a string) I figured I could just replace the "i" with an "s" and then do p.read(128). This was incorrect. After a lot of pondering over error messages I carefully read through the struct module documentation and found that you have to precede the "s" with the number of characters to be read, like "128s". So that resulted in this line...
s.Comment = unpack('128s', (p.read(128)))
...and all was well.
Remember I said that unpack() returns a tuple, and in our case the first element of that tuple is that actual integer or character array we asked for from the file? Getting the first element out of a tuple is a lot like getting the first element out of a C array. If I have a tuple called MyTuple I can get the first element by asking for MyTuple[0]. So the print lines...
print('Magic Number: %s') % hex(s.MagicNumber[0])
print('Version: %d') % s.Version[0]
print('Comment: %s') % s.Comment[0]
...do exactly that. Oh - the first line says hex(s.MagicNumber[0]) because I want the integer returned to be printed out as a hexadecimal number.
All said and done that dozen lines of code took about an hour, which isn't bad considering I started out with only a superficial knowledge of how to read bytes. Hopefully the next step of reading the more important data from the file won't be so traumatic now.
One of the biggest successes I had while learning C was reading text and binary data from files. I'm going to tackle the same task in Python. The first step is going to be reading in some data (the first 136 bytes) of the file header.
The file I'm reading is the same sort of file I learned to read in C - a file with some header data with various things like the date it was acquired and some comments, and then large chunks of digitized analog data (like a .wav file in a way).
I'm starting small - I only want to read in the first 136 bytes of the file. The first 4 bytes represent an integer that is always in the file (sort of a marker that tells us where it came from). The next 4 bytes is the version of the file format (also an integer), and the next 128 bytes represent a string of 128 characters (which are 1 byte each, so 128 characters).
I've spent a good deal of time prepping for this task - most of the info I needed was in the documentation for file objects and the struct module. In short, I'm going to read in a known number of bytes using the read() method for file objects, and then "unpack" those bytes into a specific format (integers and character strings) using the unpack() function in the struct module. So, here's a start:
I once read somewhere that using a class full of empty instance (self) variables is a good way to mimic how structures in C look. I don't know if that's very "Pythonic" but it works for me. In the PHeader class I've defined three variables that I'm going to fill in with data from the file. The actual code that executes is under "if __name__ == '__main__'", which is just a fancy Python way of saying "if this .py file is run on its own then do what's underneath".
First I open the file as "p", and initialize "s" as an instance of PHeader. I know that p.read(N) will read in N bytes of the file, so I need to somehow tell Python to interpret those four bytes as an integer (as opposed to another data type that is 4 bytes) and then make s.MagicNumber equal that resulting number.
So that's where the struct module's unpack() function comes in. unpack() has this prototype: unpack(fmt, string). fmt is the format of the bytes being read (we want an integer so we pass it "i") and string is the bytes to unpack. Well, the result of p.read(4) is our string, so this line...
s.MagicNumber = unpack('i', (p.read(4)))
...gets our four bytes, interprets that as an integer, and passes the result to s.MagicNumber. A big caveat that I missed while writing this that caused a great deal of confusing is that it doesn't ACTUALLY pass JUST the integer. It passes a Python data type called a tuple with the integer I wanted as the first element of that tuple. Tuples (and other Python data types) work a lot like arrays in other languages - but more on that in a second.
The next line does pretty much the same thing...
s.Version = unpack('i', (p.read(4)))
Ok cool, we have now read two integers from the file and stored them in some variables. This next part is tricky and caused a lot of wailing and gnashing of teeth on my part. The struct documentation told me that "i" is the format character for integers, and "s" is the format character for character arrays (strings). Since the next 128 bytes of the file is a row of 128 characters (a string) I figured I could just replace the "i" with an "s" and then do p.read(128). This was incorrect. After a lot of pondering over error messages I carefully read through the struct module documentation and found that you have to precede the "s" with the number of characters to be read, like "128s". So that resulted in this line...
s.Comment = unpack('128s', (p.read(128)))
...and all was well.
Remember I said that unpack() returns a tuple, and in our case the first element of that tuple is that actual integer or character array we asked for from the file? Getting the first element out of a tuple is a lot like getting the first element out of a C array. If I have a tuple called MyTuple I can get the first element by asking for MyTuple[0]. So the print lines...
print('Magic Number: %s') % hex(s.MagicNumber[0])
print('Version: %d') % s.Version[0]
print('Comment: %s') % s.Comment[0]
...do exactly that. Oh - the first line says hex(s.MagicNumber[0]) because I want the integer returned to be printed out as a hexadecimal number.
All said and done that dozen lines of code took about an hour, which isn't bad considering I started out with only a superficial knowledge of how to read bytes. Hopefully the next step of reading the more important data from the file won't be so traumatic now.
Tuesday, February 7, 2012
Off Topic: Gaming
I remember the first video game I played only vaguely - some text adventure. The strong memories are of DOS games like Bumpy and Continuum that we had on the 286 my dad brought home. I didn't know back then that having a computer was anything special. It was just a natural thing, and learning to use to use it came naturally. I must have read the GW-BASIC manual cover to cover a dozen times (not understanding most of it, naturally). One day my dad brought home a laptop - the first I had ever seen in person. It was massive and even for the time slow, but somehow it ran a demo of Spear of Destiny - the first 3d first person shooter I had ever seen, and to this day the most magical gaming experience I've ever had. Ever since then I've been chasing that dragon. Doom, Quake, Unreal, Half-Life, all of them. Love them. Can't get enough. The games I've fallen in love with have been pretty typical of most gamers. Quake, Half-life, Bioshock, and whatnot. All still hold a special place, and all I still play through at least once a year. I just finished my yearly run of HL2 and Portal.
MMOs are a different beast. My first MMO was Ultima Online, but not in the traditional sense. By the time I got into it a few folks had figured out how to emulate the servers, so I cut my teeth on The Alter Realm UO shard, which ran Sphere Server. I played that for years. I still jump into UO from time to time, but not on TAR since that went belly up about seven years ago. In Por Ylem is pretty good, but again - chasing the dragon.
Since UO I've tried (the official versions of) Dark Age of Camelot, World of Warcraft, EVE, Anarchy Online, Age of Conan, and Star Wars: Galaxies. I played Galaxies for about three years and loved it - the best crafting of any MMO to date. It's a damned shame what happened to it, and I'll never trust SOE with a game again. Jeez, I'm still bitter about that.
I recently started playing Star Wars: The Old Republic. I liked the Knights of the Old Republic series. Good stuff all around. It's translated to MMO format fairly well, but I have some complaints (as per tradition of all MMO players). First is the crafting. It's asinine. You just gather resources and click a recipe to craft. There is no personal influence at all, and every Widget A you craft is identical in every way to everyone else's Widget A. Yes, you can sometimes roll a 20 and make a Superior Widget A, but it's still the same as everyone else. The second is that this game has somehow driven people to hit the max level as soon as humanely possible. Everyone is either level 12 or level 50. This worries me because I'm not leveling very fast (I'm a filthy casual player) and all the patches are heavy on the end-game content that I won't see for a long time. Third is that so much of the missions are the same old "kill X space-rats". But that's MMOs for you. I'm enjoying the attention to detail in the worlds. Balmorra has wrecked ships sprinkled in the landscape. It's quite fascinating. Will I keep it up after a few months? It depends on how social I get. Right now I'm playing very solo due to not having a regular schedule. I might roll an alt to play some different storyline, but unless some major reworks to the crafting, space, and in-game market are in the pipeline I'll lose interest by the summer.
I'm not sure how I started this post. I think I've been irritated at myself for owning a lot of games I haven't played all the way through. I need to finish Skyrim. I loved Oblivion. Skyrim is fantastic, but it's turning into a chore. Battlefield 3 was a day 1 purchase, but it's the first game I've ever played that's made me feel old. I just can't compete with the current generation of gamers. Mass Effect was good. I'm about 90% through ME2, just gotta buckle down and knock out the last few parts. Same with Bioshock 2. I don't play Team Fortress 2 anymore - too many new things to keep up with.
The last game I really sank into was Deus Ex: Human Revolution. I likely won't play through it again, but it was a good time.
Ok, I'm going to bed. I could write about games all night.
MMOs are a different beast. My first MMO was Ultima Online, but not in the traditional sense. By the time I got into it a few folks had figured out how to emulate the servers, so I cut my teeth on The Alter Realm UO shard, which ran Sphere Server. I played that for years. I still jump into UO from time to time, but not on TAR since that went belly up about seven years ago. In Por Ylem is pretty good, but again - chasing the dragon.
Since UO I've tried (the official versions of) Dark Age of Camelot, World of Warcraft, EVE, Anarchy Online, Age of Conan, and Star Wars: Galaxies. I played Galaxies for about three years and loved it - the best crafting of any MMO to date. It's a damned shame what happened to it, and I'll never trust SOE with a game again. Jeez, I'm still bitter about that.
I recently started playing Star Wars: The Old Republic. I liked the Knights of the Old Republic series. Good stuff all around. It's translated to MMO format fairly well, but I have some complaints (as per tradition of all MMO players). First is the crafting. It's asinine. You just gather resources and click a recipe to craft. There is no personal influence at all, and every Widget A you craft is identical in every way to everyone else's Widget A. Yes, you can sometimes roll a 20 and make a Superior Widget A, but it's still the same as everyone else. The second is that this game has somehow driven people to hit the max level as soon as humanely possible. Everyone is either level 12 or level 50. This worries me because I'm not leveling very fast (I'm a filthy casual player) and all the patches are heavy on the end-game content that I won't see for a long time. Third is that so much of the missions are the same old "kill X space-rats". But that's MMOs for you. I'm enjoying the attention to detail in the worlds. Balmorra has wrecked ships sprinkled in the landscape. It's quite fascinating. Will I keep it up after a few months? It depends on how social I get. Right now I'm playing very solo due to not having a regular schedule. I might roll an alt to play some different storyline, but unless some major reworks to the crafting, space, and in-game market are in the pipeline I'll lose interest by the summer.
I'm not sure how I started this post. I think I've been irritated at myself for owning a lot of games I haven't played all the way through. I need to finish Skyrim. I loved Oblivion. Skyrim is fantastic, but it's turning into a chore. Battlefield 3 was a day 1 purchase, but it's the first game I've ever played that's made me feel old. I just can't compete with the current generation of gamers. Mass Effect was good. I'm about 90% through ME2, just gotta buckle down and knock out the last few parts. Same with Bioshock 2. I don't play Team Fortress 2 anymore - too many new things to keep up with.
The last game I really sank into was Deus Ex: Human Revolution. I likely won't play through it again, but it was a good time.
Ok, I'm going to bed. I could write about games all night.
Monday, February 6, 2012
First useful OOP
I finally made a functional, useful class.
I've made functional classes before, but they were only for show. This is the first time I've used OOP that actually makes programming the rest of the application easier and more organized. I'm not saying I'm a believer in OOP for the sake of OOP, but it works for me here.
Creating a new instance of this class is easy - you have to pass it a legit .csv file (no checks yet, those are coming) and on creation it chucks the data in the .csv to a list, where each list element is a type dict with key/values that match the columns/values per row in the .csv. This alone took me for freaking ever to make work because it took me for freaking ever to get my head around these Python types.
Almost every method in the Cases class is about whittling down the internally stored list of cases. Calling CasesBySalesman(salesman) alters the internally stored list to just those by a particular salesman.
I labored heavily over how to do this in the most efficient way possible. The first incarnation of this class had each method return a new list, leaving the original untouched. This, I felt, left too much work to the program that would be using this class to handle. My knowledge of the "spirit" of OOP is limited since my experience is limited, but my favorite definition so far of a class is "data, and methods to perform actions on that data", so I went with that. There is a convenient reset method to go back to baseline if necessary.
So, in the eventual program that will use this class it would be as simple as:
A = Cases(file.csv)
A.CasesBySalesman("Bob")
A.OpenCases()
And A.caselist would be a list of dicts with the open cases for Bob. This seems very straightforward. My first pass mentioned above would have involved something like:
A = Cases(file.csv)
allcases = A.AllCases()
casesbybob = A.CasesBySalesman(allcases, "Bob")
casesbybobopen = A.OpenCases(casesbybobopen)
So yea, too many variables, and here I've reduced my Cases class to just a fancy holder of functions, instead of encapsulated methods to perform actions on internal data.
I think I made the right choice, and I'm quite happy with it.
EDIT: There is a copy/paste problem in the code above, but it's not important.
I've made functional classes before, but they were only for show. This is the first time I've used OOP that actually makes programming the rest of the application easier and more organized. I'm not saying I'm a believer in OOP for the sake of OOP, but it works for me here.
Creating a new instance of this class is easy - you have to pass it a legit .csv file (no checks yet, those are coming) and on creation it chucks the data in the .csv to a list, where each list element is a type dict with key/values that match the columns/values per row in the .csv. This alone took me for freaking ever to make work because it took me for freaking ever to get my head around these Python types.
Almost every method in the Cases class is about whittling down the internally stored list of cases. Calling CasesBySalesman(salesman) alters the internally stored list to just those by a particular salesman.
I labored heavily over how to do this in the most efficient way possible. The first incarnation of this class had each method return a new list, leaving the original untouched. This, I felt, left too much work to the program that would be using this class to handle. My knowledge of the "spirit" of OOP is limited since my experience is limited, but my favorite definition so far of a class is "data, and methods to perform actions on that data", so I went with that. There is a convenient reset method to go back to baseline if necessary.
So, in the eventual program that will use this class it would be as simple as:
A = Cases(file.csv)
A.CasesBySalesman("Bob")
A.OpenCases()
And A.caselist would be a list of dicts with the open cases for Bob. This seems very straightforward. My first pass mentioned above would have involved something like:
A = Cases(file.csv)
allcases = A.AllCases()
casesbybob = A.CasesBySalesman(allcases, "Bob")
casesbybobopen = A.OpenCases(casesbybobopen)
So yea, too many variables, and here I've reduced my Cases class to just a fancy holder of functions, instead of encapsulated methods to perform actions on internal data.
I think I made the right choice, and I'm quite happy with it.
EDIT: There is a copy/paste problem in the code above, but it's not important.
Clever way of reading data into a dict type
https://github.com/breuderink/eegtools/blob/master/eegtools/io/edfplus.py
I like what's in the edf_header() function. I'm going to experiment with doing that.
Every time I noodle through GitHub I learn something new.
EDIT: The BaseEDFReader class has some justification for something I had to do to make something work - you pass the class init method a file and then it's assigning that file name to a self variable. I was struggling with why it was necessary for my stuff to work so this is evidence it's a thing you're supposed to do... for some reason.
I like what's in the edf_header() function. I'm going to experiment with doing that.
Every time I noodle through GitHub I learn something new.
EDIT: The BaseEDFReader class has some justification for something I had to do to make something work - you pass the class init method a file and then it's assigning that file name to a self variable. I was struggling with why it was necessary for my stuff to work so this is evidence it's a thing you're supposed to do... for some reason.
Friday, February 3, 2012
Gist test
GitHub has this thing called "gist" that is sort of like Pastebin:
Might use this if it behaves better than Pastebin.
Might use this if it behaves better than Pastebin.
Push
I cleaned up my GitHub account and made a repository for my hacked together but very functional security camera program.
https://github.com/cheydrick/Security-Camera
I'll be making a few changes to it soon, and I'd like to take this opportunity to get to know Git (and GitHub) better. The first change is to make it more camera and save location agnostic. Those options should be set via command line. Even better I'd like it to auto-locate a local Dropbox folder as a default if no explicit save location is made.
I got a new computer at work so I took the time to proper set up a Python development environment using Eclipse and the PyDev plugin. I'm shuffling my old support case reporting code into a more organized structure. One big change is that I'm trying to use proper classes instead of a big list of functions. Now the big list of functions are a big list of class methods, so maybe I can hang out with the cool OOP kids now, I dunno.
The new Python books are coming in handy. I'm not reading them from start to finish - just using them as a reference.
I feel super bad about neglecting C so far this year. My compromise is that I want to revisit some experiments I ran last summer where I compiled a library in C that I could call from Python. I did the usual "Hello World" stuff, but I want to explore how to create dynamically allocated arrays of stuff in C in a way that Python can then see that data.
I left my C learning progress at a good stopping point. Getting my head around OOP in Python first should help when I revisit Objective-C. Uh, Obj-C and I got in a fight and that's why I walked away from it to cool off. That's a whole different story.
https://github.com/cheydrick/Security-Camera
I'll be making a few changes to it soon, and I'd like to take this opportunity to get to know Git (and GitHub) better. The first change is to make it more camera and save location agnostic. Those options should be set via command line. Even better I'd like it to auto-locate a local Dropbox folder as a default if no explicit save location is made.
I got a new computer at work so I took the time to proper set up a Python development environment using Eclipse and the PyDev plugin. I'm shuffling my old support case reporting code into a more organized structure. One big change is that I'm trying to use proper classes instead of a big list of functions. Now the big list of functions are a big list of class methods, so maybe I can hang out with the cool OOP kids now, I dunno.
The new Python books are coming in handy. I'm not reading them from start to finish - just using them as a reference.
I feel super bad about neglecting C so far this year. My compromise is that I want to revisit some experiments I ran last summer where I compiled a library in C that I could call from Python. I did the usual "Hello World" stuff, but I want to explore how to create dynamically allocated arrays of stuff in C in a way that Python can then see that data.
I left my C learning progress at a good stopping point. Getting my head around OOP in Python first should help when I revisit Objective-C. Uh, Obj-C and I got in a fight and that's why I walked away from it to cool off. That's a whole different story.
Subscribe to:
Posts (Atom)