Using console jQuery to scrape lists from Apple’s developer portal.

Scrape
I needed to grab the lists of registered devices and developers from our company’s Apple Developer portal. Unless I’m being particularly obtuse (an outcome that I never rule out), Apple does not provide any means of exporting the lists.

Apple only allows 100 devices of each type (iPhone, iPad, iWhatever) to be registered as development devices. No matter how many iOS developers that you have at your company, 100 is the limit. And if you remove a device from that list, it still counts towards that total.  Once a year, you can reset the list and carry over the devices that you still need and drop off the ones that are not needed.  To make this easier to manage, I wanted to get a list of the devices and their ids and have the developers pick the ones that they still need.

So I wanted to export that list.  And Apple doesn’t let you export that list.  You can see it on the screen and work with the items in the list, but no export.  I figured that I wasn’t the only person dealing with that limitation so I did a quick search on Stack Overflow and found this little gem.

var ids = ["Device ID"];
var names = ["Device Name"];
$("td[aria-describedby=grid-table_name]").each(function(){
    names.push($(this).html());
});
$("td[aria-describedby=grid-table_deviceNumber]").each(function(){
    ids.push($(this).html());
});

var output = "";
for (var index = 0; index < ids.length; index++) {
    output += ids[index] + "\t" + names[index] + "\n";
}
console.log(output);

To use that code, you would go to the list of devices in the browser. Then open up the developer tools for that browser. For example, in Chrome you would press F12 to open up the developer tools. Staying with the Chrome example, you would click on the Console tab in the developer tools and paste that Javascript code in and then press the Enter key. The code would execute within the domain of the page and generate a two column list of device ids and names.

To understand what that code does, you need to look at how the data is rendered on the page. The device list is stored in a HTML table, with each row looking like this

<tr id="1" tabindex="-1" role="row" class="ui-widget-content jqgrow ui-row-ltr">
    <td role="gridcell" style="text-align:center;display:none;width: 34px;" aria-describedby="grid-table_cb">
        <input role="checkbox" type="checkbox" id="jqg_grid-table_1" class="cbox" name="jqg_grid-table_1">
    </td>
    <td role="gridcell" style="" class="ui-ellipsis bold" title="iPad" aria-describedby="grid-table_name">iPad</td>
    <td role="gridcell" style="display:none;" class="ui-ellipsis" title="c" aria-describedby="grid-table_status">c</td>
    <td role="gridcell" style="" class="ui-ellipsis" title="twletpb659m0ju078namuy8xnv2j0fzt1kytanfz" aria-describedby="grid-table_deviceNumber">twletpb659m0ju078namuy8xnv2j0fzt1kytanfz</td>
</tr>

Looking at the highlighted lines 6 and 9, we can see the device name and device id as the text of table cell tag. Each cell has a aria-describedby attribute to identity the type of value being stored. We can search on the values of the attributes to locate the data that we want. Going back to the javascript, look at the following lines:

var names = ["Device Name"];
$("td[aria-describedby=grid-table_name]").each(function(){
    names.push($(this).html());
});

The first line declares a Javascript array with an initial array element of “Device Name”. The next line performs a jQuery select for all of the <td/> elements that have attribute of aria-describedby with the value grid-table_name. The next part of the statement iterates over the list of matching <td/> elements and uses the jQuery html() to get the text value of the cell and add it to the array. We then can then do the same technique to get the device id and then build a list as a string and finally dump it to the browser’s console.

I also needed to the email addresses of all of our registered developers. The email addresses were not in a table, but part of a list. Each email address is wrapped inside a section element like this

<section class="col-100 ng-scope">
  <p ng-bind="::person.fullName" class="ng-binding">First Last</p>
  <a class="smaller ng-binding" 
    ng-bind="::person.email" 
    ng-href="mailto:first.last@yourcompany.com" 
    href="mailto:first.last@yourcompany.com">
    first.last@yourcompany.com
  </a>
</section>

I just needed the text part from the <a/> element. Getting the email addresses was a simpler version of the code to get the devices. I just a jQuery select on the ng-bind attribute and matched on the value “::person.email”. That ended up being a single line of code to run in the browser’s developer console

$('a[ng-bind="::person.email"]').each(function(){
  console.log($(this).text())
  });

And that’s how you can screen scrape data from a web page that doesn’t provide any support for exporting the data.

Bonus round
The aria-describedby attribute is a commonly used accessibility element used to describe the element that the tag is part of. The “aria” part of the attribute name is an acronym for Accessible Rich Internet Applications. Among other things, it was designed to allow assisted reading devices help parse a page for users with visual difficulties. It’s a good technology to use on your web pages.

Xamarin Dev Days – Latham, NY – Dec 2nd

Looking to start doing mobile app development with Xamarin, but don’t know where to start?  Then we have some good news for you.  Xamarin Dev Days is coming to the Tech Valley.  We’ll be hosting the event on Saturday, December 2nd, at the new Latham office of Tyler Technologies.  While it’s early to announce an event that is 9 months off, it’s still good to get the word out.

Xamarin Dev Days are community driven, hands-on learning experiences geared towards beginner mobile developers to help build, test, and connect native iOS, Android, and Windows apps.  We’ll spend the morning with sessions that introduce Xamarin ecosystem.  This will include an overview of Xamarin, Xamarin.Forms, and using cloud computing through Azure with Xamarin.

There will be a hands on lab in the afternoon that will walk everyone through how to build a Xamarin.Forms app that pulls data down from an Azure hosted database.

Agenda

Time Session
9:00 AM – 9:30 AM Registration
9:30 AM – 10:10 AM Introduction to Xamarin
10:20 AM – 11:00 AM Cross Platform UI with Xamarin.Forms
11:10 AM – 11:50 AM Connected Apps with Azure
12:00 PM – 1:00 PM Lunch
1:00 PM – 4:00 PM File -> New App Workshop

What is Xamarin?  Xamarin lets you deliver native Android, iOS, Mac, and Windows applications using your existing .NET skills and code.  You can build 100% native apps from a shared code base.  If you can do it in Swift, Objective-C, or Java you can do it in C# with Xamarin.

Tickets to this event are free, but you will need to register in advance.  Visit the Latham Xamarin Dev Days page and then click the register button.

If December is too long to wait, check out the other locations on the Xamarin Dev Days home page.  If you want to host your own Dev Days event, then click here.

A Xamarin port of the usb-serial-for-android library

Back in January, I ported the excellent usb-serial-for-android library from the Java source code to Xamarin C#.  We have an Android application that needs to use an external RFID reader.  The reader is an Elatec TWN4 RFID reader and it can work as virtual comm port over USB. To use that reader, I needed a general purpose serial over USB library.  I ended taking a very good one from the open source Java community and porting it over to C#. That ported library is up on Github under the name UsbSerialForAndroid.

Out of the box, Android doesn’t come with a lot of support for serial port devices.  It’s probably not a common use case.  Starting in Android 3.1, support was added for USB Host mode to allow access to USB devices from Android apps.  There was enough of a need for serial devices that Mike Waverly wrote a very good library in Java named usb-serial-for-android.  It supports many of the common USB serial chipsets.  So I wanted to use that.

With Xamarin Android, you have basically two ways of consuming Java libraries.  You can use them directly by creating a C# to Java wrapper and bundling the .jar file with your project.  While that can work, and work very well, it can also be a bit clunky and you can hit some issues mapping the Java method calls to C#.  Another group had gone down that path.  They implemented a wrapper for the .jar file and added some helper classes.  It looked like their project was abandonware and was not using a current version of Mike’s code.  You would also have the limitation of not being to debug into that code library.

If you have the source code, you can port the code from Java to C#.  I decided to go down that route.  It took a couple of days, but I was able to port all of the Java code to C#.  It went over more or less as is.  Some changes needed to made because reflection is handled differently in C# than in Java.  There were also a bug in Xamarin’s API access code that mangled the array handling in some Java code.

In Java, ByteBuffer.wrap(someBuffer) allows for two-way updating of a Java array with a stream buffer,  A bug in Xamarin’s API mapping tool emits code that allocates a new buffer when you call Wrap.  Changes made to the ByteBuffer are not reflected in the original byte array.  This is logged in Xamarin’s Bugzilla database here and here.

In the CdcAcmSerialPort.Read() method, defined here in C# and here in Java, I needed to add a line to copy the new array back over the original array.

In the original Java (edited) code, we had this
final ByteBuffer buf = ByteBuffer.wrap(dest);
if (!request.queue(buf, dest.length)) {
throw new IOException("Error queueing request.");
}

final int nread = buf.position();
if (nread > 0) {
return nread;
}

In the C# code, I added a call to BlockCopy to overwrite the original byte array with the updated contents
ByteBuffer buf = ByteBuffer.Wrap(dest);
if (!request.Queue(buf, dest.Length))
{
throw new IOException("Error queueing request.");
}

int nread = buf.Position();
if (nread > 0)
System.Buffer.BlockCopy(buf.ToByteArray(), 0, dest, 0, dest.Length);
return nread;
}

I also replaced some integer constants with enumerated types where it made sense to do so. I also took the C# helpers from the LusoVU repository.

As much as I could do so, I followed the code structure’s of the Java library.  When changes are made to that library, I can view those changes and make the equivalent changes in the C# code.  The end result was that I ended up with all C# code and it works great.

The TWN4 has become my favorite RFID reader.  It’s very good at reading different card types and you can write custom firmware for it in C.  I used it in another project where it had to work with a custom protocol to with the host device.

TWN4 reader