News | PostgreSQL for Mac | Andy Satori

PGSQLKit for iOS

Mon, 26 Jul 2010 20:46:41 -0400

Today we make available something new(ish). As we've posted in the past, we have been tinkering with PGSQLKit as an iOS library for accessing remote PostgreSQL servers. Please bear in mind, this is not the whole stack, it is just a client access library at this point.

In the spirit of being forthright about everything, we seriously considered holding this internally and making it a commercial product. Like most small businesses right now, we need the revenue. The problem is finding a balance between Open Source and revenue generation is proving to be a difficult one. This years total revenue to date from PostgreSQL (advertising and donations) is $126.58.

That said, in the spirit of what we want of this project we could not in good faith keep this as a commercial project. So here we go. As of today, PGSQLKit for iOS is part of the project.

In this first revision, the setup and deployment is still a pain in the arse. It will get better as we refine the process. Currently in the zip is a set of three other zips. ipsql.zip is nothing more than a sample of a simple iOS project that is built upon PGSQLKit. It shows how the files are included and used. The second is PGSQLKit.zip. This archive is the SVN tree of the PGSQLKit code, the needed files will need to be added to your project from here. Finally is the biggie, pqtouch.zip. In this archive is the xcode project to build libpq from the PostgreSQL sources for iOS. An entire PostgreSQL source tree exists in the folder, as there is some work in progress at getting PostgreSQL itself running on iOS. That is work that will continue, but is a VERY low priority.

When we update the mainline distribution to PostgreSQL 9, this too will be updated.

Good luck and have fun coding.

9.0 beta 3, 8.4.4 and 7.4.29

Mon, 19 Jul 2010 10:16:21 -0400

On the agenda for this week is getting the packages updated for the new versions. We are starting with the oldest and working forward. For the most part this is fairly straightforward work, but after the problems with the .pkg from 8.4.3, we have to do some testing and debugging on the packages that may take more time than we expect.

PGSQLKit, iPad, PostgreSQL9

Tue, 06 Jul 2010 14:51:12 -0400

Tackling a few subjects here, so this might be a bigger post than usual.

First, there has been some work with PGSQLKit over the last couple of months that we need to talk about. While much of the work has revolved around getting everything working so that the iPhone / iPad can be used as clients to a PostgreSQL database, there has also been a good bit of work to address several other issues. One that comes up fairly often is the ability to store binary data. Though we haven't finished the PreparedStatement work (we still haven't found a solution for this we really like), we did add a pair of new functions to the PGSQLConnection object to make it easier to manage binary data.

-(NSData *)sqlDecodeData:(NSData *)toDecode;

-(NSString *)sqlEncodeData:(NSData *)toEncode;

These two functions work in tandem to encode and decode binary data for use in sql commands. While storing large amounts of data in binary form is not really what PostgreSQL is designed for, it works well for small amounts of binary data.

In addition, all of the objects have been altered to allow for better support of string encodings. For PostgreSQL9, we are moving towards using a local platform mapping to determine a default encoding based upon your locale. Most programs that use PGSQLKit will need to call setDefaultEncoding: to ensure correct string encoding operations as the defaults will change to NSUTF8String with PostgreSQL9.

iPad / iOS

The iPad and iOS4 platforms present some interesting issues for us. At this point, everything in PGSQLKit but PGSQLLogin works on iOS3 and iOS4. However, because frameworks are not supported, the classes have to be imported as source. We hope to have a sample project uploaded in the coming months to show how all of this works.

The flip side to this however, is the server. We get frequent requests to run PostgreSQL itself on the iPad. Though in our testing, it is something that *could* be done we are not entirely sure if that translates to *should*. First, battery life would be impacted. Second, for single user needs, is this necessarily a good answer? Third, assuming that it was all configured and running, what are the real applications the PostgreSQL engine on the iPad?

At this point, we do not have good answers, and so this part of the project will, for the immediate future at least, remain more of a theoretical / proof of concept project. If we see a compelling reason to change this, we will.

PostgreSQL9

With PostgreSQL9 having recently moved to Beta2 stages, we are expecting a release sooner than later. This means that we are gearing up to deliver a new installer, that hopefully addresses most of the weaknesses of the current approach. At this point, we think we will have something ready to go about 4 weeks after the official PostgreSQL9 release, but as always, the paying work will take precedence.

This weekend :(

Fri, 30 Apr 2010 12:06:53 -0400

Unfortunately, we are in a tough spot for the weekend. As most of you know, I (dru) do that vast majority of the work in the project. Unfortunately, my primary machine suffered a major failure last night. My Mid 2009 MacBook Pro apparently cooked it's GPU. So it is now a glorified paperweight that requires a new logic board.

Obviously, that leaves me without a mobile to work from home on.

Though I have shipped the machine to Apple this morning, I do not know how soon I will get it back, nor am I entirely certain I will. It is *very* close to the 'not in warranty' time in it's lifecycle. If it is not warranted, then I will be faced with a nasty choice. $$$ to repair or $$$$ to replace, both at a time when things are a little lean around the checkbook parts (having just purchased iPads for development purposes earlier this month). Given that, the work towards 8.4.3 will be heavily impacted this weekend and well into next week.

I will try to keep this updated as I have more information. If we get the predicted rain, I may come in o the office and work here over the weekend to get this moving.

Time Away to Rethink Installers, PostgreSQL

Tue, 27 Apr 2010 17:29:45 -0400

First let me apologize for dropping off the face of the earth for the last 3-4 weeks. Paying work got a little nuts and then we ran into our annual user group meetings for those same customers. Between those, that pretty much killed a month.

That said, it really turned out to be a net positive, because it gave me a bit of time away from the current state of affairs to really sort out where we need to go with this. To really gnaw on some of the core issues that surround this project in specific, but also PostgreSQL as a whole. It also let me sort out where I think this project needs to go with regards to those issues.

Earlier this year, I created a PostgreSQL steering committee. Ultimately, they will help make some of these decisions, but I am going to lay out my thoughts here and now, and we can all take it from there. First we need to tackle the issues (as I see them).

The Issues

Lack of external tracking of issues

The strength of PostgreSQL may well be one of it's greatest weaknesses. The distributed 'work on what you want' nature of Open Source development means that sometimes issues can sit for a long time before they get addressed. To make matters worse, PostgreSQL has elected not to use an issue tracking system like Bugzilla, but has instead chosen to use a mailing list for tracking bugs. This can make it tricky to filter out what is or is not being worked on. I know of a couple of issues that I have elected not to work on because I believed them to be in process based upon -bugs and -hackers mail traffic. One has existed for a while and has not been addressed. I know that those effected are looking at alternative platforms to PostgreSQL because they have no way of knowing if an issue is being worked on.

This means that people that are not active on both -bugs and -hackers have no way to 'track' or 'watch' issues once they have been reported. It also means that some of us that are not full-time on the PostgreSQL mailing lists sometimes have difficulty monitoring the status of a particular issue. I do not see this changing within the PostgreSQL.org environment.

Granularity of Focus

This is not a PostgreSQL issue so much as it is an issue with this project. The initial goal and mindset of this project was very narrow. Build PostgreSQL with the most common options and configure it so that it fit naturally into the Mac OS environment and ecosystem. I think we accomplished that, and then the scope grew into the GUI tools, and then the developers frameworks. Somewhere along the we, that narrow focus either got lost or misplaced.

Apple's Installer is not well suited to the PostgreSQL for Mac Project

Do not misunderstand, the Installer.app is a nice tool, but for a project that has so many updates, and such a massive base of potential add on modules, it is not well suited to the need. What is needed is something that has more in common with the package managers or Sparkle than Installer.app.

No central repository for the modules and addons for PostgreSQL

This is an old problem. There is a repository in pgFoundry, if you build from source. You can also use DarwinPorts or Fink to install everything, but there are issues with all of the above. Not the least is that most servers do not have the Developer Tools installed on them. It also means an additional learning curve for many people that are not prepared to, or are outright afraid to tackle.

Dealing with the Issues

There is no silver bullet here, and honestly we are a very small fish in the pond, even in the smaller Mac Using PostgreSQL pond. So the scope of what we *can* tackle is relatively limited. Acknowledging that, it occurs to us that what we can do is tackle each of these topics in our little corner of the pond.

Lack of external tracking of issues

Unfortunately, there are limits to what issues we can tackle in the trunk code base. What we can tackle though, we can track. Though I am not the worlds biggest fan of SourceForge, that is where we set up shop for PostgreSQLforMac 6 years ago. It makes no sense to go elsewhere at this point, despite my own preference for a JIRa/Confluence/Bamboo configuration. We can start tracking issues that we are either 'tracking' on -bugs and -hackers or are working internally. This gives our customers a place to go o track issues that have been reported directly to us.

(Francois, if you would resubmit your history on your Collation bug, we think we would like to start with that one)

We know that this will never percolate to the larger PostgreSQL community, the discussions there regarding bug tracking systems are just not worth rehashing. Suffice it to say, they will choose to do what they choose to do.

Granularity of Focus

This is really a tough one. The truth is, we would really like to be a one stop shop for all things PostgreSQL on the Mac. This would require a time commitment that our PostgreSQL derived income simply cannot support. Understanding this, we think that solving the next issue will actually help with this while making the real desire a little bit more attainable. In the short term though, the focus almost has to go back to the original goal, along with a smaller subset of the tools to make it all work. The other fluff pretty much has to go back being paid for work until the total revenue is sufficient to support it. Sadly, it comes back tot he 'making money on Open Source is tough' comments that you are seeing in mainstream tech media right now.

Apple's Installer is not well suited to the PostgreSQL for Mac Project & No central repository for the modules and addons for PostgreSQL

This one, and the next one are tightly wound together, so we are 'bundling' them together. The solution to one, quite naturally links to the other.

The more we fight with Installer.app and how it overwrites directories, the more evident it becomes that Installer.app just doesn't work for what we want to do. Before we get into the details, we need to give some background.

If you have looked inside the /PostgreSQL8/ folder lately, you will have noticed that it's internal structure is built to support multiple versions in the same folder that can be 'switched' at the users discretion. This idea is to allow developers and deployers both to easily switch between minor versions without a major hassle of backup/uninstall/reinstall/restore. We would really like to also make it trivial to do the same with the various 'addon' modules that PostgreSQL supports (think PostGIS as a 'for instance').

We have spent months trying to work within the limitations of Installer.app to accomplish this, and yet the harder we push it, the worse it behaves.

For that last couple of months, we have been tossing around the idea of doing something else for the future installs. The 8.4.3 installer failures have only pushed this idea harder. After a few weeks away, I am sold.

What we want to do is add a PostgreSQL Installation Manager to the distribution. Using it, you can install a PostgreSQL version from a locally stored package, or from a Server based package that we provide. The process works like this:

A. User downloads the 'unified installer'. It installs the installation manager and a small collection of local packages (current version, gui tools, etc). Once complete, it allows the user to select add ons that may be wanted, directly from the PostgreSQL for Mac Repository. When a new version is released, it works like Sparkle, enabling a self update, while retaining your current version. You can then choose to 'test' a new version and roll back quickly if it doesn't work out. When you are done with a version, you can easily remove it.

Yes, this is a ton of work. In the long run though, it provides a blueprint for something that can be the centerpiece of the best Mac Database solution on the market, and that is what we want.

But what does that mean in the short term? well, it means that a fixed 8.4.3 has to be dealt with. Probably this weekend. It also mean that 9.0 is the earliest we would see this. There are a couple of other changes that have to be dealt with for 9.0 anyways, but that is a topic for another book :).

Regardless, we are not going to tackle this until we get some thoughts from you the users, so please post up, or email me. We need to work on some design stuff, at this point everything is pie in the sky concept stuff.

For users with 8.4.3 already installer

Mon, 29 Mar 2010 20:36:47 -0400

There is a quick way to get an 8.4.3 installation that failed working. Open Terminal.app and run the following:

cd /Library/PostgreSQL8/bin

sudo ln -s postgres postmaster

cd ../lib

sudo ln -s libecpg.6.1.dylib libecpg.6.dylib

sudo ln -s libecpg.6.1.dylib libecpg.dylib

sudo ln -s libecpg_compat.3.1.dylib libecpg_compat.3.dylib

sudo ln -s libecpg_compat.3.1.dylib libecpg_compat.dylib

sudo ln -s libpgtypes.3.1.dylib libpgtypes.3.dylib

sudo ln -s libpgtypes.3.1.dylib libpgtypes.dylib

sudo ln -s libpq.5.2.dylib libpq.5.dylib

sudo ln -s libpq.5.2.dylib libpq.dylib

exit

That will recreate the missing files from the distro.

Issues with 8.4.3, Resolution incoming

Mon, 29 Mar 2010 11:12:12 -0400

The 8.4.3 release over the weekend has a major issue. Somehow, the symlinks are being dropped from the 8.4.3 installation. All of the files are there, but not the links. This is causing failures in launching and using.

Now that we know the issue we've got a fix in process and hope to have the package updated later today.

Released 8.4.3 and 8.3.10

Sat, 27 Mar 2010 19:03:50 -0400

Both of the updated installers are now available. These are just updates to the core DB with few other changes.

One caveat though. There is an issue in the installers that rather than leave a current version in place, the installer overwrites them. We are not entirely sure how to fix the problem, but there are a couple of options we are exploring. If you need the prior version to stay put, please copy it aside and then copy it back in after the installation.

8.4.3 building now; release this week

Mon, 22 Mar 2010 17:56:51 -0400

Just a quick note that with the weekend release of 8.4.3 and 8.3.10 we are already in process with these releases to update the distro. Expect a release this week, but it means that we will not be ready with nQuery for the iPad before the app store cutoff for the launch.

Back with more as we have news.

Another week, another update

Thu, 18 Mar 2010 11:45:02 -0400

So this week has seen a handful of additional changes to PGSQLKit. PGSQLKit 8.4.2c has been published with this posting. As we work forward with getting our iPhoneOS work done, we have been spending a lot of time in PGSQLKit (as it is what we are using in the iPhoneOS world). That time has seen us finding little issues but also time to make some needed tweaks.

Most of our work right now is subtle changes like replacing NSMutableStrings that aren't with NSStrings, or identifying release/retain pairs that are not properly matched. Some of the work though is addressing issues raised. If you have been following some of the discussions in the comments of the recent posts, you'll see that we have had the string encoding bugbear rise again.

String Encoding

We addressed this in a simple form not too long ago, but it was just that, a simple quick answer. This update addresses the situation in a more complete way, and lays the groundwork for the future with regards to future usage. The downside is that people using the PGSQLKit today that expect a certain behavior will need to adjust to something new when PostgreSQL 9 is available.

The fundamental change is that effective with 8.4.2c, PGSQLKit now implements a defaultEncoding on all of the objects, starting at the PGSQLConnection object. The default is NSMacOSRomanStringEncoding to remain consistent with the old behavior. Please note, THIS WILL CHANGE WITH PostgreSQL 9.

If you change the defaultEncoding before opening a PGSQLRecordset, all of the string operations on will adopt the new default. Existing PGSQLRecordsets will not be effected, but can be changed by setting their respective defaultEncodings.

What this means is that if defaultEncoding is changed, then all string operations will use the new encoding unless explicitly overriden in cases like using PGSQLFields asString:(NSStringEncoding)encoding method. It also means that using dictionaryFromRecord will return strings using the defaultEncoding.

The defaultEncoding can be altered at any level and always cascades down: Connection -> Recordset -> Record -> Field.

This solution should provide enough power and flexability to deal with most any encoding issues in the future.

Now about the change:

With PostgreSQL 9, the default value for defaultEncoding will change from NSMacOSRomanStringEncoding to NSUTF8StringEncoding.

Documentation

As we continue the work, we are expanding the headerDoc documentation. This documentation is included in the PGSQLKit download, and will eventually find it to the website. The only reason it is not there now is that we have not had the time (or inclination to be honest) to sit down with a CSS editor to really clean up the layouts. It will happen, it just hasn't yet. If someone with a flair for CSS and HeaderDoc has the time and inclination, I promise we won't complain :-).

libpqtouch?

So, unless you didn't catch it, libpqtouch is a version of libpq compiled as a static library that we are using as the foundation of PGSQLKit for the iPhoneOS. The discussion around the office is how to handle this. Right now, the project is a manually updated build of libpq that is built in Xcode to create an armv6 build of libpq. None of us have been able to get a good build of libpq directly from the command line using PostgreSQL sources without eventually hand assembling the files.

This makes it very difficult to publish the changes back to the PostgreSQL community. So, what we are proposing is this: Publish the steps and the Xcode project. Until things stabilize with our project, we are holding on to the work, but hope to get it into the public SVN as soon as we submit the iPad application to the iTunes App Store.