How to access Apple’s App Connect API from C#, Python, and Go. – Part 4

Welcome to Part 4 of a three part series. Last month, I did a series of posts on how to use Apple’s App Connect API to query the team membership list using C#, Go, and Python. This was code I was actually using myself and after I wrote it, it stopped working. The API calls were returning an HTTP 401 error. But only on Windows, on the Mac the code worked.

That was an odd one to track down. It turned out to be a bug with how I was generating the expiration timestamp for the JWT payload.  Apple documents that you can specify up to 20 minutes into the future for the exp value. Don’t do the full 20 minutes. That works most of time. Until it doesn’t. Just set the timeout to 10 minutes into the future and Bob’s your uncle. 

I went back to the repo’s and changed the expiration timeout to 10 minutes for the C#, Go, and Python repos. The C# code was set to 30 minutes, that should have never worked. If you tried the C# code and wondered why it wasn’t working, that was my mistake.

While I was updating the code, I changed the Go version so that you could validate multiple user names by placing them in a line delimited file and pass that file in with a “-userlist” command line parameter.  The code now reads all of the usernames and stores them in an array.  It then gets all of the team members and checks to see if any of the user names in the array match the team member list.  If you need to validate multiple users, this makes it a one and done task.

The “-username” parameter can still be used if you just have one user to check. In that case, it’s treated as an array with just one item in it. If you use both “-username” and “-userlist”, the code will use the user list and ignore the user name passed on the command line.

I’m not sure why the code worked when I originally wrote the posts a couple of months ago. My guess is that Apple used to allow timestamps longer than 20 minutes but made a change that now enforces the expiration to their published specification. Test for edge conditions, but avoid using them.

How to access Apple’s App Connect API from C#, Python, and Go. – Part 3

In my first post, I wrote about a need to query my company’s membership list from our Apple app store development account. In that post, I used C# to query Apple’s API. Part 2 covered the Python version. For our final installment, we’ll cover how to accomplish the same task with Go.

As with the C# and Python versions, we’ll need to create a signed Javascript Web Token (JWT) and then make some API calls.  I have the code in a Github repository. You can clone it from here. As with the C# project, the code is spread over a couple of modules. 

The code should be portable across platforms. I used Go 1.13.2 on Windows. If you don’t have Go installed, you can grab it from the download page at golang.org. This would be a good place to mention that the only experience that I have with Go is what I learned from writing this applet. I’m pretty comfortable with C#, but everything that I know about Python and Go came from writing this code.

That disclaimer in place, after you install Go, you can grab the code from the repository and follow along. As with the other projects, we use a 3rd party library to generate and sign the Javascript Web Token (JWT). When I wrote the code a couple of months ago, I used dgrijalva’s jwt-go library. It’s no longer being maintained and it recomends to use a community supported clone of that project, golang-jwt/jwt. While writing this blog post, I updated the code to use the community supported project.

If you clone the code from my repo, the jwt library should get downloaded when you build the code. If it doesn’t you can install it the following command
go get github.com/golang-jwt/jwt

Before we run the code, lets take a tour of the code. The entry point is the module IsUserInApple.go. You can follow along with the code from the repo. It will differ from the C# and Python versions by having named command line parameters. We start with the following code

package main

import (
	"flag"
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	// Define command line options
	configPtr := flag.String("config", "./IsUserInApple.json", "Configuration file")
	usernamePtr := flag.String("username", "", "Username to find (in quotes)")

	flag.Parse()

	var userName string = *usernamePtr
	var ConfigFileName string = *configPtr

At line 4, we import the flag module. This module makes it simple to define and read command line parameters. We’ll define 2 parameters, one for the name of the config file (line 12), the second for the name to match (line 13). For our code, we’ll define them as string variables. We pass in the name of the command line parameter, default value, and the help text.

After defining the parameters, we call flag.Parse() to parse the command line parameters that are being passed in. Lines 17-18 assign the flag values to string variables. This code could probably be written with less code, but it works and it’s readable. The next bit of code checks to see if a username was passed. If it doesn’t, it prints the command line help and then quietly dies a good death.

if len(userName) == 0 {
	fmt.Println("Please specify an email address to match (in quotes)")
	flag.PrintDefaults()
	os.Exit(1)
}

The next few lines check to see if a config file was specified.

if len(ConfigFileName) == 0 {
	ex, err := os.Executable()
	if err != nil {
		panic(err)
	}
	ConfigFileName = filepath.Join(filepath.Dir(ex), "IsUserInApple.json")
}

if _, err := os.Stat(ConfigFileName); os.IsNotExist(err) {
	fmt.Println(err)
	os.Exit(2)
}

At line 1, we see if a file has been specified. It hasn’t been specified, then we want to look in the folder that code is running from from for a file named IsUserInApple.json.  The os.Executable() method will return the full path for the running code, or an error message if something went wrong. If that happens, we abandon ship with panic().

If you are new to Go, it’s a little wierd to see both a return value and error message on the left side of a function call. But you get used to it after a while. And it’s a very clean way of getting errors back from a method call. After getting the path, we extract the directory and concatenate the name of the config file.  At line 9, we check to see if the file exists or die trying.

At this point we have a filename for the configuration file and we can attempt to read it.

fmt.Println("Looking for " + userName)

config, err := ReadConfig(ConfigFileName)

Time to dive into the ReadConfig() method. That code is in the AppleJWT.go module. It’s not the cleanest separation of code, but just roll with it. We have a bunch of fun things going on here.  First things first are the imports:

import (
	"crypto/ecdsa"
	"crypto/x509"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"io/ioutil"
	"log"
	"time"

	"github.com/golang-jwt/jwt"
)

The first 8 imports are modules that come with Go. The crypto modules are used for the JWT signing. The encoding modules let us read and parse JSON and PEM data. The last module is the external library that will be pulled directly from Github.

Now we’ll define an object to contain the JSON data from the config file. This is a pretty simple object:

type ConfigSettings struct {
	PrivateKeyFile string `json:"PrivateKeyFile"`
	KeyID          string `json:"KeyID"`
	IssuerID       string `json:"IssuerID"`
}

That will map to the JSON format of the config file, which will look like this

{
    "PrivateKeyFile": "path/to.your/privatekey.p8",
    "KeyID": "ABCDEF1234",
    "IssuerID": "d88b7c23-4c26-48fb-9d62-5649f27a25a2"
}

There are some handy online tools from converting JSON data to Go structs. I have used JSON-to-Go and transform.tools, they both work pretty well. The JSON-to-Go tool gives you the option of using inline type definitions or separate structs for each type. After that we have the ReadConfig method

func ReadConfig(ConfigFileName string) (*ConfigSettings, error) {
	file, err := ioutil.ReadFile(ConfigFileName)

	if err != nil {
		return nil, err
	}

	config := new(ConfigSettings)

	err = json.Unmarshal([]byte(file), &config)

	return config, err
}

Line 1 defines the parameters, the name of the config file, and an error variable. Then we read the file at line 2 with the ReadFile method. If it fails, we return the error. At line 8, we create a new instance of the ConfigSettings struct that will contain our settings. Then at line 10, we use the Unmarshal() method to parse the JSON data into our ConfigSettings struct.

That brings us back to our main method. 

if err != nil {
	fmt.Println(err)
	os.Exit(3)
}

CheckUserList(config, userName)

Skipping over error checking code, we now have the method that will ties this all together for us. Now we can dive in to CheckUserList() defined in AppleStoreApi.go. At the top of module, we have some structs defined.

type AppConnectUsers = struct {
	Data  []Datum              `json:"data"`
	Links AppConnectUsersLinks `json:"links"`
}

type Datum = struct {
	Type       string     `json:"type"`
	Attributes Attributes `json:"attributes"`
}

type Attributes = struct {
	Username  string   `json:"username"`
	FirstName string   `json:"firstName"`
	LastName  string   `json:"lastName"`
	Roles     []string `json:"roles"`
}

type AppConnectUsersLinks = struct {
	Self string `json:"self"`
	Next string `json:"next"`
}

type AppConnectErrors struct {
	Errors []struct {
		Status string `json:"status"`
		Code   string `json:"code"`
		Title  string `json:"title"`
		Detail string `json:"detail"`
	} `json:"errors"`
}

The AppConnectUsers, Datum, Attributes, and AppConnectUsersLinks structs define the JSON data that is returned by the List Users API call. If you look back at Part 1, the JSON document returned by List Users has a lot more in it. We only need to define structs to use the data members that we actually care about. Since this is a read only API call, we can let Unmarshall deserialize only the fields that we will use and it will discard the rest.

We have a second object called AppConnectErrors, with a very different structure. If the API call fails from something happening on Apple’s end, the API will send back a very different JSON document. The following is an example of the JSON that Apple sends back when the API is not available:

{
	"errors": [{
		"status": "500",
		"code": "UNEXPECTED_ERROR",
		"title": "An unexpected error occurred.",
		"detail": "An unexpected error occurred on the server side. If this issue continues, contact us at https://developer.apple.com/contact/."
	}]
}

We’ll cover how to handle that error data later on. The CheckUserList method has a lot going on so we’ll do this in parts.

func CheckUserList(config *ConfigSettings, Username string) {
	token, err := CreateAppleJWT(config)
	if err != nil {
		fmt.Println(err)
	}

	client := &http.Client{}

	var nextUrl string = "https://api.appstoreconnect.apple.com/v1/users?limit=100"

The first thing is that we call CreateAppleJWT in AppleJWT.go to create the signed JWT. Now we’ll jump into that.

func CreateAppleJWT(settings *ConfigSettings) (string, error) {
	bytes, err := ioutil.ReadFile(settings.PrivateKeyFile)

	if err != nil {
		fmt.Println(err)
	}

	x509Encoded, _ := pem.Decode(bytes)

	parsedKey, err := x509.ParsePKCS8PrivateKey(x509Encoded.Bytes)

	if err != nil {
		log.Fatal(err)
	}

	ecdsaPrivateKey, ok := parsedKey.(*ecdsa.PrivateKey)

	if !ok {
		panic("not ecdsa private key")
	}

	token := jwt.NewWithClaims(jwt.SigningMethodES256, jwt.MapClaims{
		"iss": settings.IssuerID,
		"exp": time.Now().Add(time.Minute * 20).Unix(),
		"aud": "appstoreconnect-v1",
	})

	token.Header["kid"] = settings.KeyID

	tokenString, err := token.SignedString(ecdsaPrivateKey)

	if err != nil {
		log.Fatal(err)
	}

	return tokenString, nil
}

At line 2, we read the private key file. Lines 8 through 16 parse the PEM data and come out with the private key in a format that we can use for signing. At line 22, we create the token with an expiration timestamp of 20 minutes into the future. We set the token header kid field to our KeyID. We sign the token at line 30 with our private key and pass it back. Getting the token signed correctly was the hardest part of writing this code. Getting back to CheckUserList…

client := &http.Client{}

var nextUrl string = "https://api.appstoreconnect.apple.com/v1/users?limit=100"

var FoundMatch = false

for {
	req, err := http.NewRequest("GET", nextUrl, nil)
	if err != nil {
		fmt.Print(err.Error())
		os.Exit(3)
	}

	req.Header.Add("Authorization", "Bearer "+token)

	resp, err := client.Do(req)
	if err != nil {
		log.Println("Error on response.\n[ERROR] -", err)
		os.Exit(3)
	}

	defer resp.Body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Println("Error while reading the response bytes:", err)
		os.Exit(3)
	}

We start with a new instance of an http Client. We initialize nextUrl with the initial API url. And we set FoundMatch to false. Now we start looping. At line 8, we new up a new request and add our token at line 14. Line 16 is where we call the API and we wait at line 22 to get the response back.

We read the entire body into a byte array at line 24. Now we can start with the parsing. The first thing to do is see if the API returned an error document instead of the list of users.

var appConnectErrors AppConnectErrors
err = json.Unmarshal(body, &appConnectErrors)
if err != nil {
	log.Println("Error while deserializing the response bytes:", err)
	os.Exit(3)
}

// If there is an error object in the body, print it and exit
if len(appConnectErrors.Errors) > 0 {
	firstError := appConnectErrors.Errors[0]
	log.Println("Status:", firstError.Status)
	log.Println("Error accessing API:", firstError.Detail)
	os.Exit(4)
}

We new up an instance the AppConnectError struct as appConnectErrors. Then we unmarshal the body into appConnectErrors. Next we check the length of the Errors array In appConnectErrors. If the API returns users, the Errors object will not be there and the length will be 0.

If the length of Errors is greater than 0, then “Houston, we have a problem“. While Errors is an array and could have more than one error, we don’t care about anything after the first error. So we get the first (and probably only) error. Then we log the error status, which will be a numeric code, and the full text message. And then we die with an error code of 4.

Otherwise we keep going…

var appConnectUsers AppConnectUsers

err = json.Unmarshal(body, &appConnectUsers)
if err != nil {
	log.Println("Error while deserializing the response bytes:", err)
	os.Exit(3)
}

for _, s := range appConnectUsers.Data {
	FoundMatch = strings.EqualFold(s.Attributes.Username, Username)

	if FoundMatch {
		fmt.Printf("Found %s, %s %s, %s\n",
			s.Attributes.Username,
			s.Attributes.FirstName,
			s.Attributes.LastName,
			strings.Join(s.Attributes.Roles, ", "))
		break
	}
}

if FoundMatch {
	break
}

nextUrl = appConnectUsers.Links.Next

if len(nextUrl) == 0 {
	break
}

As with the error struct, we new up a AppConnectUsers struct named appConnectUsers. We unmarshal the JSON data into the struct. At line 9, we iterate through the list of users stored in the Data array. We set FoundMatch at line 10 to true if the username matches. If we have a match, we print that match to the console and use break to exit the for..range loop

Next we break out of the forever loop if we have a match. Otherwise we check to see if we have a next link. If we do, we keep looping, otherwise we are done looping. And we end with an message if we didn’t have a match. 

If you cloned the code from the repo and created your own config file (refer back to Part 1 for that part), you can now compile and run the code. You would compile the applet from your terminal of choice with the following command:

go build .

If you want the smallest executable, build it with the following options:

go build -ldflags="-s -w" .

And then you can compress the resulting executable with a tool like upx. You would need to run IsUserInApple with a required command line parameter, the email to look for. If you leave it out, you will get an error message telling you what is needed. Remember that the IsUserInApple.json file needs to be in the same folder as the executable if you don’t use the second optional command line parameter that will let you specify the name of the config file.

If you are running it with the go command, you would run it as

go run . -username "some.email@company.com"

or

go run . -username "some.email@company.com" -config "IsUserInApple.json"

If you are running it as the compiled executable, you would run it as 

IsUserInApple -username "some.email@company.com

You’ll back something like this:

Looking for some.email@company.com
Found some.email@company.com, Some Email, APP_MANAGER

or….

Looking for some.email@company.com
No match for some.email@company.com

On the rare situation that the API is not available and Apple returns an error document, you’ll see something like this:

Looking for some.email@company.com
2021/07/13 23:26:19 Status: 500
2021/07/13 23:26:19 Error accessing API: An unexpected error occurred on the server side. If this issue continues, contact us at https://developer.apple.com/contact/.
exit status 4

If you run it without  the -username parameter or pass is “-h” or “–help”, you will get back the usuage help messages

go run .                                                                  
Please specify an email address to match (in quotes)
  -config string
        Configuration file (default "./IsUserInApple.json")
  -username string
        Username to find (in quotes)

go run . -h
Usage of C:\Users\anoth\AppData\Local\Temp\go-build2936985627\b001\exe\IsUserInApple.exe:
  -config string
        Configuration file (default "./IsUserInApple.json")
  -username string
        Username to find (in quotes)

That wraps up this series. You can do more than get the team list from the App Connect API. For example, if you want to track how many devices have been provisioned, there is a Devices API. Lots of access points for DevOps functionality.

How to access Apple’s App Connect API from C#, Python, and Go. – Part 2

In my previous post, I wrote about a need to query my company’s membership list from our Apple app store development account. In that post, I used C# to query Apple’s API.  In this installment, we’ll cover how to accomplish the same task with Python.  The final post will cover a Go version.

As with the C# version, we’ll need to create a signed Javascript Web Token (JWT) and then make some API calls.  As with the C# version, I have the code in a Github repository. You can clone it from here. The code is in a script named IsUserInApple.py. You can name your script anything that the OS and Python allows, I’ll be referring to my script as IsUserInApple.py.

I wrote the code using Python 3, version 3.9.2. Unless you are running Windows, the odds are that you have Python 3 installed. If not, you can get it from the good people at python.org. This code should work on any platform that supports Python 3. It should also work (potentially with some changes) with Python 2, but I haven’t tested that. 

This script will use a couple of libraries that you will need to install. To make the HTTPS web requests, we’ll be using the Requests library. It does exactly what it says on the tin, “Requests is an elegant and simple HTTP library for Python, built for human beings”. It makes the code for calling Apple’ API simple and easy to follow.

The other library is Authlib, a library for working with OAuth and OpenID Connect. It has everything we need to create and sign our JWT.  While we need to install all of Authlib, we’ll only being using the jwt module.  To install the libraries, you just need to run the pip command like this:

pip install requests authlib

If you are not familar with pip. it’s the package manager for Python and will be installed when you install Python. If you already have Python installed, then you already have pip installed.

In the folder that you will have the Python script, you will need a configuration file. It performs the same function as the IsUserInApple.json file did in the C# code. It will contain the path to your private key file, the key id, and the issuer id for your account. Please refer back to Part 1 for how to define these values. Instead of using JSON, I used a simple key/value file and it’s named IsUserInApple.config. It should use the following format:

[settings]
private_key = c:/scripts/AuthKey.p8
KEY_ID = ABCDEF1234
ISSUER_ID = d88b7c23-4c26-48fb-9d62-5649f27a25a2

Now we can go through the code. I’m going to jump around a bit, you may want to have the script open in another window. After some comments, we start with the following lines:

import requests, time, json, sys, tempfile, os, configparser
from authlib.jose import jwt

The first line imports the libraries that we’ll need. The requests library was the 3rd party library that we installed via pip, the rest are libraries included with Python. The next line imports the jwt model from the authlib.jose library. The “jose” part of the library name is an acronym for Javascript Object Signing and Encryption.

The next part of the code are the methods we’ll define and use to call the API. We’ll dive into them in a bit. For now, we’ll jump down to the bottom of the script.  We start with

if len(sys.argv) > 1:
    config = configparser.ConfigParser()
    try:
        configPath = os.path.dirname(os.path.abspath(__file__)) + "/IsUserInApple.config"
        config.read(configPath)
    except Exception as e :
        print(str(e))

Line 1 is saying “if we have at least one command line parameter after the name of the script…”. The next line creates an instance of the ConfigParser class, which will allow us to easily read contents from the IsUserInApple.config configuation file. Starting at line 3, we’ll use a try/except block to read the config file. We want to read the config file from the same folder as the script. To get that folder name, we call the abspath method on the __file__ variable. The __file__variable is a “dunder” variable in Python and represents the name of the currently running module. A list of dunder variables can be found in the Python Docs. The dirname method will return the folder part of the file name and then we concatenate that with the name of the config file. And at line 5, we read the config file.

Next, we read the settings into variables with the following code. It’s pretty much self explanatory.

try:
    private_key = config['settings']['private_key']    
    KEY_ID = config['settings']['KEY_ID']    
    ISSUER_ID = config['settings']['ISSUER_ID']    
except Exception as e :
    print(str(e))

Next, we have the following code:

if not os.path.isfile(private_key):
    sys.exit("Error missing private key file for JWT")

UserEmail = sys.argv[1].lower()

print('Looking for a match on ' + UserEmail)

token = getToken(KEY_ID, ISSUER_ID, private_key)

First we check to see if the value provided for private_key is actually a file. Then we set UserEmail to the command line parameter (while converting it to lowercase) and echo that back to the shell. The last line is where we call our getToken method in the script to generate the signed JWT. Now, we’ll jump to that getToken method.

def getToken(KEY_ID, ISSUER_ID, PATH_TO_KEY):
    EXPIRATION_TIME = int(round(time.time() + (20.0 * 60.0))) # 20 minutes timestamp

    with open(PATH_TO_KEY, 'r') as f:
        PRIVATE_KEY = f.read()

    header = {
        "alg": "ES256",
        "kid": KEY_ID,
        "typ": "JWT"
    }

    payload = {
        "iss": ISSUER_ID,
        "exp": EXPIRATION_TIME,
        "aud": "appstoreconnect-v1"
    }

    # Create and return the JWT
    return jwt.encode(header, payload, PRIVATE_KEY)

Line 1 defines the name of the method and the parameter names. No big whoop. Line 2 generates the expiration time, 20 minutes into the future. At line 4, we read in the private key file. The jwt module groks the PEM format, we don’t have to clean it up like we did in the C# code. Then line 7 creates the header and line 13 creates the payload. This is nearly identical to the code from the C# version. At line 20, we create and sign the JWT and return to the code that called it. It follows the same logic as the code in the GetToken method from AppleJWT.cs. Jumping back to where we called getTokem, we have the following line:

members = getAllUsers(token)

Now we’ll dive into the getAllUsers method.

def getAllUsers(token):
    JWT = 'Bearer ' + token.decode()
    URL = 'https://api.appstoreconnect.apple.com/v1/users?limit=100'
    HEAD = {'Authorization': JWT}

    teamMembers = []

    nextURL = URL
    keepGoing = True

    while keepGoing:
        r = requests.get(nextURL, params={}, headers=HEAD)

        y = r.json()

        if 'errors' in y:
            errorCode = y['errors'][0]
            print('Apple returned an HTTP ' + errorCode['status'] + ' code')
            print(errorCode['detail'])
            sys.exit('whoops')

        for i in y['data']:
            teamMembers.append({'username':i['attributes']['username'].lower(), 'roles': ','.join(i['attributes']['roles'])})

        if 'next' in y['links']:
            nextURL = y['links']['next']
        else:
             keepGoing = False

    return teamMembers

Line 1 defines the method name and parameters. Lines 2-4 sets the bearer (token) authorization and the inital URL to call. The next few lines just initialize a few variables and the fun begins at line 11 where we’ll loop until we are done. At line 12, we use the requests library to make a HTTPS get call. Line 14 assigns the JSON results to our Y variable. The next block of code checks the JSON document for “errors”. If Apple’s API falls down, it will fall down this way.

If there are no errors, we just walk through the “data” member of JSON document and add the username and roles for each user to the teamMembers array. When we append the username, we convert it to lowercase. That was we don’t have to worry about case when we match on the email address. Then we check the see if the “links” member of the document has a field named “next”. If it does, we use that as the new URL and keep looping. When we don’t have a “next” url, we return the teamMembers array to caller. That takes us back to the bottom of our script again.

    for i in members:
        if i['username'] == UserEmail:
            HasMatch = True
            print('Match on ' + i['username'] + ', Roles: ' + i['roles'])

    if HasMatch == False:
        print('No match')

else:
    sys.exit("Error: Please specify an email address")

Now we just iterate through the members array and see if the user name matches. We report back on whether we foiund it or not.  The last lines are the error message that would be reported if you ran the script without a email address to match on.

That’s basically all there is to this script. This version doesn’t deserialize the JSON data into objects, it just parses it as is. It’s a trade off. Doing it this way uses less code, but you lose some of the discoverability of having the data in object form.

To run it, you would just do something like:

python IsUserInApple.py some.email@company.com

You’ll either the following if that use is a member of the account:

Looking for a match on some.email@company.com
Match on some.email@company.com, Roles: ADMIN,ACCOUNT_HOLDER

Or the following if that email doesn’t provide a match

Looking for a match on some.email@company.com
No match

Coming up next will be the Go version of this script.


About the image:
This image was derived from some open-source images. The image of the wall comes from Patrick Tomasso via Unsplash. The Apple lock logo comes from Apple, which retains all rights to its artwork. The people icon was created by Monika from the Noun Project.

How to access Apple’s App Connect API from C#, Python, and Go. – Part 1

As the Account Holder for my employer’s Apple App Store account, I get to keep track of who has access to the account. Apple does not do federated logins. Which means no linkage between our Active Directory store and the user accounts associated with the App Store account.

So if someone leaves our company, their Apple Dev account stays active on our team account until we go in and manually remove it. When you have thousands of employees, there’s no way for a developer like me to know who has left the company. And I shouldn’t know that, that’s why we have HR to manage those resources and IT to handle the offboarding.

To make it easier for IT to manage this, I wrote an applet that they could run to see if an employee had an account on on our App Store team. They have a limited API that you can query to get information about your apps and team members. Basically you run that applet and pass in the email address of the employee. It will come back and tell you if that address is a member of the account. 

I had written code in C# that I would run from LINQpad to query various App Connect Services in an ad-hoc manner. I made a stand alone version for our IT department and then realized they would need to have the .NET Framework installed. So I decide to port the functionality to Python and Go (aka “Golang”). Installing the Python runtime is easy and with Go, I can just create a single executable. I’m going to split this blog post into 3 parts, each part covering a different language. This will be the C# version. Part 2 covers Python. The final post in the series, Part 3, covers a Go implementation..

This version of the applet was written for .NET 5. I wrote and tested the code with Windows 10, but it should compile and run under Linux and MacOS as long as the .NET 5 SDK has been installed. All of the source code can be cloned from this repository on Github.

To query Apple for information about your development account, you need to use their App Store Connect API. It’s more or less (I’ve cover that in a bit) documented here.

To authenticate to their API, you’ll need to generate a signed JSON Web Token (JWT). You’ll sign the key with a private key that will be generated on Apple’s site. They’ll hold the public key, you’ll have the private key. This is managed from Apple’s API portal and you can revoke a private key at any time.

You’ll need to generate an API key through your App Store Connect account. The JWT needs to be signed with that key or Apple will reject it.

It’s pretty easy to generate a key. Apple lists the steps here:

  1. Log in to App Store Connect
  2. Select Users and Access, and then select the API Keys tab.
  3. Click Generate API Key or the Add (+) button.
  4. Enter a name for the key. 
  5. Under Access, select the role for the key.
  6. Click Generate.

You’ll be able to download the API private key once as a .p8 file, Apple does not store your private key. You’ll want to store it securely. If you lose it, you’ll have to revoke it and generate a new one.

The API call that we want is List Users. List Users will return all of the users along with their associated metadata. By default, it will the first 100 users, you can increase that up to 200 with the limit parameter. If there are more users than can be supplied in the request, the payload will include a link to get the next set of records. You would keep calling the “next set” URL until it stops including a “next set” URL. 

The JSON shown below is a subset of a fake result set that could be returned by a call to List Users. You get a set of users and the information that we need are fields in the “attributes” object, highlighted below:

{
  "data" : [ {
    "type" : "users",
    "id" : "bded051a-7566-4b5f-a7a9-2e461e51eab0",
    "attributes" : {
      "username" : "smithj@fake.com",
      "firstName" : "John",
      "lastName" : "Smith",
      "roles" : [ "APP_MANAGER" ],
      "allAppsVisible" : false,
      "provisioningAllowed" : true
    },
    "relationships" : {
      "visibleApps" : {
        "links" : {
          "self" : "https://api.appstoreconnect.apple.com/v1/users/bded051a-7566-4b5f-a7a9-2e461e51eab0/relationships/visibleApps",
          "related" : "https://api.appstoreconnect.apple.com/v1/users/bded051a-7566-4b5f-a7a9-2e461e51eab0/visibleApps"
        }
      }
    },
    "links" : {
      "self" : "https://api.appstoreconnect.apple.com/v1/users/bded051a-7566-4b5f-a7a9-2e461e51eab0"
    }
  } ],
  "links" : {
    "self" : "https://api.appstoreconnect.apple.com/v1/users?limit=50",
    "next" : "https://api.appstoreconnect.apple.com/v1/users?cursor=SomeHideousToken&limit=50"
  },
  "meta" : {
    "paging" : {
      "total" : 150,
      "limit" : 50
    }
  }
}

For our purposes, we only care about the email address. We walk through the JSON and just add the address to a list. If there is a “next set” URL, we call that to get the next set of addresses. Then we can compare those emails with the email addresses of the former employees to see which accounts will need to be removed.

This “Next” set URL really isn’t documented as part of the API call. This caused me a fair amount of angst. The API documentation states that the maximum number of users returned will be 200. We have 150+ on the team and I converned about what would happen when we passed 200 members. I emailed Apple Support and was told that API works and I would get all of the members. But they didn’t mention how.

I did some searching in Apple’s forums and the solution was simple and logical. In the JSON result set, there is “links” object. The links object will have a “self” field that contains the URL that was used to make the call. It can have an optional “next” field that will contain a URL that will return the next set of data. When you call the API, you will need to check the “next” field and call that URL until you no longer receive another “next” field in the JSON result set. 

"links" : {
    "self" : "https://api.appstoreconnect.apple.com/v1/users?limit=50",
    "next" : "https://api.appstoreconnect.apple.com/v1/users?cursor=SomeHideousToken&limit=50"
}

Depending on your needs, you can either parse the data in each set or combine it into one set. It (hint) should documented in a way that would be easier to discover. So getting back to the data, we want the “username” from the “attributes” object. The name and role data is nice to have, so we’ll grab it.

"attributes" : {
  "username" : "smithj@fake.com",
  "firstName" : "John",
  "lastName" : "Smith",
  "roles" : [ "APP_MANAGER" ],
  "allAppsVisible" : false,
  "provisioningAllowed" : true
}

The actual API key data is stored in a separate file that I created named IsUserinApple.json. It will contain the path to your private key file, the key id, and the issuer id for your account. It will look something like this:

{
    "PrivateKeyFile": "path/to.your/privatekey.p8",
    "KeyID": "ABCDEF1234",
    "IssuerID": "d88b7c23-4c26-48fb-9d62-5649f27a25a2"
}

That file will not be in the repo, you’ll have to create that one yourself. To get the data, we need to make the API call. We need to pass in a signed JWT. This will be the heaviest lifting for the code. There are three steps:

  1. Create the JWT header
  2. Create the JWT payload
  3. Sign the JWT

For the .NET version, we are going to use a nuget package, jose-jwt, to create and sign the JWT. In the repo for this version of the applet, the code for generated the signed JWT is AppleJWT.cs. The full method is short and we can quickly go over what it does here.

/// <summary>
/// Returns a signed JSON Web Token
/// </summary>
/// <param name="keyId">Your private key ID from App Store Connect</param>
/// <param name="issuerID">Your issuer ID from the API Keys page in App Store Connect</param>
/// <param name="privateKey">The private that was generated by Apple, encoded as Base64</param>
/// <returns>Signed JWT</returns>
public string GetToken(string keyId, string issuerID, string privateKey)
{
    // Create the header
    var header = new Dictionary<string, object>()
    {
        {"alg", "ES256"},
        {"kid", keyId},
        {"typ", "JWT"},
    };

    // Create the payload
    var exp = Math.Round((DateTime.UtcNow.AddMinutes(30) - new DateTime(1970, 1, 1, 0, 0, 0)).TotalSeconds, 0);

    var payload = new Dictionary<string, object>()
    {
        {"aud", "appstoreconnect-v1"},
        { "exp", exp },
        { "iss", issuerID }
    };

    // Generate the signing key from the private that has been given to us
    CngKey key = CngKey.Import(Convert.FromBase64String(privateKey), CngKeyBlobFormat.Pkcs8PrivateBlob);
    
    // Generate the signed token
    string token = Jose.JWT.Encode(payload, key, JwsAlgorithm.ES256, header);

    return token;
}

Lines 11-16 create the header. We set the expiration date at line 19 to 30 minutes into the future with the “exp” field. The issuer ID at line 25 comes from your account in App Store Connect. Apple’s instructions for getting the issuer ID are fairly easy to follow: log in to App Store Connect and: Select Users and Access, then Select the API Keys tab. The issuer ID appears near the top of the page. To copy the issuer ID, click Copy next to the ID. The key ID will also come from that page. It will be the key that we described how to create earlier.

The ID values that you will need

The private key will come from the contents of the .p8 file that you downloaded when you created the API key. The .p8 file will have your private key in PEM format. We generate the signing key from the private key at line 29 and then generate a JWT and sign it with that key at line 32.

There is some code in program.cs to read the PEM data in the .p8 file and return just the private key needed to sign the JWT. The .p8 file will contain something vaguely like this:

-----BEGIN PRIVATE KEY-----
mfykFJXw6fZBxYR4GYDGKyF27GfFaedt4tvR0jACGwUJxBR2t83wZQZOtoj6PMPC
seSrR0XXVWhEzImSXtLfF8JCyHrvpknEgOzSvqD040LUOvgNrnMvrGQHmPGQwmta
So6GaxJuCk0XxJa2t9mjfu5ERvPCIVwQPgeqcVrroNx2qaXzBteLsyonXOPRxo2w
VnPiPup3
-----END PRIVATE KEY-----

Our code needs only what exists between the “BEGIN” and “END” lines. So this code just strips all of that out

private string GetPrivateKey(ConfigSettings configSettings)
{
    var certPEM = File.ReadAllText(configSettings.PrivateKeyFile);

    return certPEM
    .Replace("-----BEGIN PRIVATE KEY-----", "")
    .Replace("\n", "")
    .Replace("\r", "")
    .Replace("-----END PRIVATE KEY-----", "");
}

Now that we have a signed JWT, we can make the API calls. In AppStoreApi.cs, there is a simple wrapper for the List User API call. This unit has a fair amount of code that provides C# models of the JSON data. I used the tools at app.quicktype.io to generate the C# models from the JSON data. I wont repeat that code here, it’s all in repo. But we’ll review the highlights. We start off with by creating a descendant of HttpClient that will be pass the signed JWT in the headers.

public HttpClient client {
    get {
        if (_client == null)
        {
            _client = new HttpClient();    
            _client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Bearer", token);
        }

        return _client;
    }
}

We’ll have a public method, FindUser. It will take an email address and return a user object. Or null, if it doesn’t find a match.

public User FindUser(string EmailAddress)
{
    var users = GetAllUsers();

    if (users != null)
    {
        var user = users.Where(s => s.UserName.Equals(EmailAddress, StringComparison.CurrentCultureIgnoreCase)).FirstOrDefault();

        return user;
    }

    return null;
}

The API call comes in GetAllUsers().

public List<User> GetAllUsers()
{
    List<User> users = new List<User>();

    // Find the first 100 users.  If there there are more than
    // 100 users, the "Next" property will contain the net URL to call
    var jsonString = GetUsers(100, null).Result;

    if (jsonString == null)
    {
        return null;
    }

    var appConnectUsers = AppConnectUsers.FromJson(jsonString);
    
    users.AddRange(appConnectUsers.Data
        .Select(s => s.Attributes)
        .Select(s => new User() {UserName = s.Username, 
                                    LastName = s.LastName, 
                                    FirstName = s.FirstName, 
                                    Roles = s.Roles.ToList()}) );
    
    while (appConnectUsers.Links.Next != null)
    {
        jsonString = GetUsers(100, appConnectUsers.Links.Next.ToString()).Result;
        appConnectUsers = AppConnectUsers.FromJson(jsonString);
        users.AddRange(appConnectUsers.Data
                .Select(s => s.Attributes)
                .Select(s => new User() { UserName = s.Username, 
                                        LastName = s.LastName, 
                                        FirstName = s.FirstName, 
                                        Roles = s.Roles.ToList() }));
    }

    return users;
}

GetUsers is the call the App Connect API and will return a JSON result set. The FromJson method is some boilerplate code for deserializing the JSON. Then we use some LINQ code to iterate through the result and pull the fields from the attributes objects. Then we continue to call the API until we no longer have a Links.Next value.

The GetUsers method is pretty straightforward.

private async Task<string> GetUsers(int count, string nextUrl)
{
    var url = nextUrl ?? $"https://api.appstoreconnect.apple.com/v1/users?limit={count}";
        
    var result = await client.GetAsync(url);
    
    if (result.StatusCode == System.Net.HttpStatusCode.InternalServerError)
    {
        Console.WriteLine(result.StatusCode.ToString());
        return null;
    }
    else
    {
        var users = result.Content.ReadAsStringAsync();

        return users.Result;
    }
}

We pass in null for nextUrl for a new search, otherwise it’s the url for the next set of data. 99.99% of the time you will get a result back. When Apple’s API is down (and I have seen it happen), you can get a 500 error and this code will catch.

Assuming that you have the .NET 5 SDK installed, you can just grab the code from the repo and then build it.  Executing “dotnet build” should pull down the required packages and the build the applet. If you are running it with the dotnet command, you would run it as 

dotnet run -- some.email@company.com

If that email is a member of the account, the applet wiill come back with something line this:

Found user: some.email@company.com, Some Email [AppManager, Developer]

Otherwise something like this:

User: some.email@company.com not found

That’s the core of the code. The rest of the code is some simple stuff for processing the command line parameters and loading the settings. It’s pretty basic stuff and commented in the repo. Our IT staff now uses a version of this code when they process an employee leaving the company. If that person’s email shows up, IT passes the email to me and I can remove that user from the account. We may look at ways to have the applet query the results from the App Connect API and then match all of the emails to see if they exist in our Active Directory. In the meantime, this works.

Coming next will be the Python and Go versions of the code.


About the image:
This image was derived from some open-source images. The image of the wall comes from Patrick Tomasso via Unsplash. The Apple lock logo comes from Apple, which retains all rights to its artwork. The people icon was created by Monika from the Noun Project.

Getting a new dev machine set up, macOS Style

It’s that time of the decade when I get a new Macbook Pro. I use a Macbook to compile iOS apps with Xamarin which makes it required equipment. Even though my current one was a late 2014 model, it was still fast enough to do what I needed to do. It’s a work laptop and was out of warranty, so I was due for a new one. The 2014 one will go to another developer.

The last time I did this, I did a blog post on what I installed. This will version will be mostly the same, plus adding the cool new things that have come down the pike in the last 4 years.

FileVault
First thing to do is to enable Apple’s full disk encryption. If someone steals this thing, they can’t access the files. They can get my body, but not my soul.

VPN
This is my work laptop. Which means the usually Corporate IT requirements. Which includes a VPN so that I can connect back to the mothership when I’m remote. Plus the corporate AV solution de jour.

Files
I had tons of files to move over. I was pretty good about keeping the files under the user folder. Between my old MacBook and this one, I used AirDrop. I also copied a fair amount of files from the old Macbook to my home server and then from the home server to the MacBook.

I took care in copying over the .ssh folder. Which in retrospect was a waste of time, It’s easier just to regenerate the public keys.

Homebrew
This is a one of those command tools that makes it easier to get other command line tools (git, node, iterm, etc). Here’s a good guide for how to install Homebrew.

Fira Code
If you spend half of your day slinging code around, treat yourself to a font that is designed for coding and has ligature support. Fira Code is a nice, easy on the eyes font and has the ligature support that your eyes have been craving.  Using ligatures with coding is a personal preference and there are some good arguments against using them.  I would suggest turning off ligature support when presenting your code or posting code listings.

Iterm2
While MacOS comes with fully capable terminal app, there are ones out there that provide a more productive workspace. One of them is Iterm2.  Better copy/paste, better tabs, better autocomplete.  Just get it from here.

Zsh
After installing iTerm2, the next thing is to install Zsh with Oh My Zsh. Zsh is an *nix shell built on, but greatly improved over Bash, the standard MacOS shell. Oh My Zsh is a an open source kit for managing Zsh. I followed this guide for installing Zsh and Oh My Zsh. And I’m learning how to use it from this cheatsheet.

QuickRes
If you are doing screen captures, you don’t always want the captures done at Retina resolution. I usually use a second monitor at 1920×1080 (1280×720 if recording for Pluralsight), but if you only have the MacBook, you can use 3rd party apps to force the resolution down. QuickRes is a popular one. I haven’t had the need to use of of these utilities, so this is more of a way of stashing that information.

Xcode
Even though I do my coding with the .NET stack using Xamarin, I still need to have Xcode installed.  Apple makes the Xcode compilers part of the tool chain.  Plus that’s the only way you get the iOS Simulator.

Certificates
The bane of any Apple developer. I have a bunch of developer certs to move over. I didn’t use any sane way of moving them. I just let Xcode pull them down to the new machine. Afterwards, I discovered Fastlane.

Fastlane is an open source project designed to manage Android and iOS deployment and it has command line tools and strategies for managing the Unholy Hell that is Apple codesigning. I think I’m going to redo how we handle certificates and provisioning profiles for the next project and use Fastlane to manage them.

Visual Studio 2019
As a Xamarin developer, this is where I spend a great of deal time in. When installing, make sure to add Xamarin Workbooks. Get it here.

After installing VS, I used to have to update ~/.android/advancedFeatures.ini to include the line
HVF = on
This sets Google’s Android Emulator to use Apple’s Hypervisor-Framework to speed up the performance of an x86 emulator image. I used to use Intel’s HAXM, but we don’t need that anymore. The Google Android Emulator can now use the native hypervisor on MacOS and Windows. Taking HAXM out means one less thing to worry about breaking when you update the OS. With the current install of VS 2019, this is set for you.

Visual Studio Code
This has replaced every other text editor for me on both Windows and the Mac. I use it edit MarkDown files, write C for embedded hardware, quickly hack .csproj files, and pretty much any file that starts with “.”. Get it here.

If you want to run VS Code from the command line by just typing code, do the following:
Open the Command Palette (⇧⌘P) and type ‘shell command’ to find the Shell Command: Install ‘code’ command in PATH command.

Docker
Remember when we used to spin VMs to run specialized apps? I still do run Windows on my Mac, but for everything else we have Docker containers. It’s 2019 and the ability to run SQL Server on a Mac in a Docker container is old news. So you’ll probably want Docker Desktop for the Mac.

Pluralsight Offline Player
I’ve been a big fan of Pluralsight for a number of years. And as of this year, I’m also a Pluralsight author. If you get the Offline Player App, you can download courses and watch them without an Internet connection. Like on a plane.

Vysor
When testing Android code, an actual device works much better than an emulator. Vysor provides a remote desktop to the Android device, which is handy for using the keyboard and doing presentations. Vysor uses ADB to make the connection, but if you doing Android development work, the odds are high that ADB has already been installed. There is a free version, but get the paid Pro license. You get more features and it keeps the product funded.

OneDrive
I share files with multiple providers and on multiple machines.  I use OneDrive a lot.  I have a personal account of about 20 GB and an Office 365b account for work.  I used to be a big fan of Dropbox, but they changed their policies and it costs more to use an account on multiple computers.  So I’m phasing out my usage of DropBox.

Microsoft Edge
Who knew that Microsoft would do a better Chrome than Google? It behaves a lot like Chrome, but it feels a little snappier. Competition is a good thing. Get it here.

Google Chrome
While the Chromium based Edge browser is my day to day browser, I still use Chrome. It has better synching of bookmarks and easy access to Google Services. You know where to find it.

Adobe Creative Cloud
Photoshop and Illustrator are still the gold standard for working with bitmap and vector images. I also use Premiere and After Effects for inhouse video projects. If you are creating animations with Lottie, you are going to need After Effects.

PaintCode
PaintCode is one of those really cool developer apps that only exist for the Mac. This app lets you take and edit vector images (.ai, .svg, .pdf) and convert them to resources that you can use in your apps. It can take an image and convert to an object in Swift, Objective-C, C#, and Java. It can also export Android Vector Drawables and SVG files.  My typical usage for PaintCode is to take a source image from Adobe Illustrator in .ai format and use it to render Android Vector images along with .png (with the @2 and @3 variants) for iOS.

A really cool feature is if you are creating an object in code, you can assign properties to parts of the images. You can move, rotate, or scale them. Change their colors, animate them, change the visibility. Very handy when you have an object that change appearance based on the input from the user or the data.

Snagit
This is probably the best screen capture tool that you can use. I use it on both Windows and Mac and I just love it. From the good people at TechSmith.

Camtasia
Also from Techsmith comes Camtasia, a video editing app that is great from online training courses. It’s what I use for making Pluralsight courses.

Office 365
The Office apps on the Mac have more or less parity with the Windows version. It’s what we use at work, so I have most of the Office experience. In additional to the standard Office apps, I add Skype for Business and Teams. Our team lives inside MS Teams and it’s pretty much the same experience on both platforms.

Slack
I’ve installed the Slack desktop app as an experiment. I have been using the web page, but when you belong to a bunch of Slack sites, it starts getting cumbersome to manage.

Postman
A handy tool for checking web service calls. Design and test your web API calls with one tool.  Get it here.

VLC
This video player will play just about anything. Video files, DVDs, streaming protocols, etc.

Google Earth Pro
Vey handy when working with KML files and just fun to play with. You can do more with the desktop edition of Google Earth than is available with the online versions.

Audacity
While Adobe CC has a good audio editor with Audition, most of the time when I need to edit an audio file, Audacity is just what I need. It’s free and open source and it does pretty much what most of us need,. I should learn Audition though.

Parallels Desktop
When I need to run Windows on my Mac, I use Parallels. I used to use VMWare Fusion, but it was just a pain in the ass. Dragging a window from a Retina screen to a FHD didn’t handle the font scaling correctly. And it was slow.

Moving my Windows 10 VM from my old Mac to the new one was trivial. I copied the VM as a file from one machine to the other. I then installed Parallels on the new machine and it was able to open the VM without any complaints. When I enter in the activation code, it prompted me to deactivate the license on the old PC. I hate having to deal with licenses, but this was tolerable.

BetterTouchTool
This should have been baked into the OS. If you want window snapping features or want to customize the Touch Bar, then get BetterTouchTool.

Disk Inventory X
Sometimes you run out of disk space. This will show you where it went

Windows only apps that I would to see on the Mac

Multilingual App Toolkit
This is a Windows only tool from Microsoft. It provides a workflow for managing language resource files and can perform machine translation through Cognitive Services. It integrates into Visual Studio, but only Windows.

Markdown Monster
This is a neat editor for editing Markdown files. While VS Code has pretty decent support for Markdown, Markdown Monster is better.

LINQPad
I have had days where this is the only development tool that I have touched. When I’m testing some new .NET code or tweaking calls to a web service, LINQPad is the tool that I tend to grab first. And I really want a Mac version.

Hardware

At my desk, I use both Windows and Mac machines. I often go back and forth. I used to have identical keyboards with separate mice. This takes a fair amount of desk space. For the last 100 years, my keyboard of choice has been the Microsoft Natural Keyboard 4000. If I’m lucky, I get a year out of one.

After I fried the last one, I decided to switch gears a bit. I got the Logitech Craft keyboard and MX Vertical Mouse. The MX Vertical supports Logitech’s Flow feature. When paired to two or three computers, you can treat the desktops as single expanded desktop. When I move the mouse off the right edge of my Windows desktop, the cursor is on the left edge of the Mac desktop. And the keyboard switches at the same time. You can also copy and paste between the computers. It’s a neat trick and probably works as expected 98% of the time.

I’ve been using the MS Natural keyboards since they came out and I’m still getting used to the regular layout. The vertical mouse helps my wrist more than then keyboard and having just one keyboard on my desk is worth the compromise. They are wireless and use USB-C, which made me like them before opening the box.

Docks
For years, I’ve been asking for every port that wasn’t USB-C to just go away. For better or worse, I got my wish with this Macbook Pro. It has 4 USB-C/Thunderbolt 3 ports, and oddly enough given what Apple did to the iPhone, a headphone jack.

My old MacBook Pro has this wonderfully insane dock from Henge Docks. You place the MacBook down and two arms would automagically connect to each side, replicating all of the ports to the back. Except some how it gained a full sized Display Port connector and Ethernet port. It was crazy and fun, but tied to the port configuration of the models that it supported. So they stopped making it.

I now have one of their new models, the Stone Pro. It sits underneath the Macbook and connects with a USB C/Thunderbolt 3 cable. When I get to work, I just connect the one cable and my external monitor and iOS devices are connected. And it has it’s own dedicated power supply. Which means I get to leave the Apple brick in my backpack.

I did get a smaller dock that connects to the side of the MacBook Pro, using two of the USB C connectors. It replicates the ports, plus adds a HDMI connector, USB A ports, and SD/Micro SD card slots. All of which is useful when I go somewhere and want to share my screen.

I do want to note that as much as I love having a chunky USB C charging brick, I already miss the Mag Safe 2 connector. There are 3rd party magnetic USB C connectors, but I have yet to see one that would fully support the power and data functionality. There is something common sense about a power cable that wont drag your laptop off the table when someone trips over the cord.

I get that it’s trickier to do data connections with a magnetic connector. A momentary disconnect of the power lines will have little to no effect. Dropping the data lines is different matter altogether. But Apple, find a way to make it work.

In the 4 years since the last Mac upgrade, it has become a lot easier to work all day on the Mac. All of the MS Office apps that I use are there and have more or less parity with the Windows versions. The developments tools are on both. Still better on Windows, but that gap is narrowing. The Mac keyboard still drives me crazy, but since I use external keyboards most of the time, it’s just something for me to complain about.

All product logos in this post are property of their respective owners.

When your Apple device refuses to be trusted

Back in July, I received a new Macbook Pro.  A lovely little device, but I had this one problem.  It wasn’t showing up as a trusted device under my iCloud account. 

When you log into a site or a service that requires Apple’s 2 factor authentication, you can use a Trusted Device to generate a 6 digit authentication code.  Most people use their iPhone, but I don’t have an iPhone.  Plenty of other Apple devices, just not the phone.  And 2 factor authentication works on my iPad, my old Macbook, and my Mac Mini.  Just not on the new Macbook. 

The 2 Factor Auth prompt

When I’m travelling, the only Apple device that I carry is the Macbook.  When I log into a service that needs the Apple auth, I want to be able to use my Macbook to provide that code.  That’s a pretty reasonable request.

This is what you should get

This has been frustrating me for a good month or so.  There is no switch or setting to enable.  Once you enable your iCloud account on an Apple device made in the last few years, that device is now a trusted device. That’s all you are supposed to have to do.

Except for my new Macbook.  It would not display the dialog that would prompt for the authentication code.  I tried Apple Support via email, but that proved to be fruitless.  I don’t think that they fully understood the problem.  I tried again today with Apple Support via phone.  It took the better part of an hour, but we finally resolved the problem.

Basically everything was setup correctly and should have worked.  But it didn’t.  With anything computer related, sometimes the best solution is to nuke the problem from a Low Earth Orbit.

It’s the only way to be sure

So we removed the iCloud account, rebooted, and added it back again.  That resolved the problem.  The Apple Support rep thinks that the iCloud auth token on the Macbook was corrupt. I have no idea how or why that happened, but that would explain what caused the problem.

Getting the most out of VMware Fusion 8.5 running Windows 10

I’ve been trying to get the most performance out of my Window 10 virtual machines running on my MacBook Pro through VMware Fusion. I have a Windows 10 virtual machine that I use for software demos and testing beta versions of Windows. It’s been running much slower than you would expect on a 2 3 year old MacBook Pro with a quad core i7.  I’ve collected the following tips (the sources are listed at the end) and they have improved the performance.

From the MacOS Side

Exclude the virtual disks from Time Machine backups.

You’ll want to avoid trying to back up the virtual machines by Time Machine.  If Time Machine is trying to back up the virtual machine while it is being used, it will probably fail to perform the backup and it will definitely throttle the disk I/O.

  • Run the Settings App
  • Open “Time Machine”
  • Click the “Options” button
  • Under the “Exclude these items from backups”, click the “+” button.
  • Select the Virtual Machines folder.  By default, this will be located in your documents folder.  Once you have selected the folder, press the “Exclude” button.
  • Press the “Save” button

If you are running an anti-virus application on your Mac, make sure that it is excluding the Virtual Machines folder

From the Virtual Machine Side

With your virtual machine stopped, you can make some system changes to achieve better performance.  Within Fusion and with the virtual machine open (but not running), open the Settings dialog.  You’ll want to make the following changes:

  • Open “Display” and clear the “Accelerate 3D Graphics” checkbox.
  • Open “Processors & Memory”
    • Set the number of processor cores to a value of n-1 or less, where n is the number of actual cores on your Mac.  My Macbook Pro has a quad core i7, so I run with 2 cores assigned to the virtual machine.
    • Give the virtual machine as much ram as you can, but without starving the host OS.  My Mac has 16 GB, so I split it 50/50.  If you have less memory, remember to leave at least 2 GB to the MacOS OS.
    • Open Advanced Options and select “Enable hypervisor applications in this virtual machine”
  • Open “Hard Disk (SCSI)”
    • Open “Advanced options”
    • Set bus type to SCSI
    • Set “Pro-allocate disk space” to enabled.

There are some settings that are not directly exposed through the settings dialog.  You’ll need to modify the .xmx file directly.  There are a couple of ways of getting at the .vmx file, the clearest technique is documented on the vmguru.com page: “Modifying the .vmx file step-by-step”.

  • Change ethernet0.virtualDev = “e1000e” to ethernet0.virtualDev = “vmxnet3”
    This will change the default network adaptive to an enhanced driver
  • Add the line scsi0:0.virtualSSD = 1
    This will optimize disk I/O for SSD drives.  Only use this if your MacBook has a SSD drive
  • mainMem.backing = “swap”
    May speed up memory swap files
  • MemTrimRate = “0”
    Disable memory trimming, less overhead for the Fusion memory manager
  • sched.mem.pshare.enable = “FALSE”
    Turns off memory sharing between virtual machines
  • prefvmx.useRecommendedLockedMemSize = “TRUE”
    Speed up I/O at the cost of increased memory usage in the host OS
  • MemAllowAutoScaleDown = “FALSE”
    Prevents Fusion from attempting to start the virtual machine with less memory than specified.  This can trigger Windows activation.
  • logging = “FALSE”
    Disabling the logging should speed things up a bit

If you don’t need snapshots, remove them.  When you use a snapshot, disk I/O is parsed through each snapshot.  That will show things down.

 

Resources for these suggestions

  1. VMware Performance Enhancing Tweaks (Over-the-Counter Solutions)
  2. Making Windows 10 inside VMWare Fusion 8.x a bit quicker on OSX 10.11 El Capitan
  3. How to Fix Slow Windows VMs on VMware Fusion 8.x
  4. Excluding the Virtual Machines folder from being backed up by Time Machine (1014046)
  5. Troubleshooting Fusion virtual machine performance for disk issues (1022625)

Using console jQuery to scrape lists from Apple’s developer portal.

Scrape
I needed to grab the lists of registered devices and developers from our company’s Apple Developer portal. Unless I’m being particularly obtuse (an outcome that I never rule out), Apple does not provide any means of exporting the lists.

Apple only allows 100 devices of each type (iPhone, iPad, iWhatever) to be registered as development devices. No matter how many iOS developers that you have at your company, 100 is the limit. And if you remove a device from that list, it still counts towards that total.  Once a year, you can reset the list and carry over the devices that you still need and drop off the ones that are not needed.  To make this easier to manage, I wanted to get a list of the devices and their ids and have the developers pick the ones that they still need.

So I wanted to export that list.  And Apple doesn’t let you export that list.  You can see it on the screen and work with the items in the list, but no export.  I figured that I wasn’t the only person dealing with that limitation so I did a quick search on Stack Overflow and found this little gem.

var ids = ["Device ID"];
var names = ["Device Name"];
$("td[aria-describedby=grid-table_name]").each(function(){
    names.push($(this).html());
});
$("td[aria-describedby=grid-table_deviceNumber]").each(function(){
    ids.push($(this).html());
});

var output = "";
for (var index = 0; index < ids.length; index++) {
    output += ids[index] + "\t" + names[index] + "\n";
}
console.log(output);

To use that code, you would go to the list of devices in the browser. Then open up the developer tools for that browser. For example, in Chrome you would press F12 to open up the developer tools. Staying with the Chrome example, you would click on the Console tab in the developer tools and paste that Javascript code in and then press the Enter key. The code would execute within the domain of the page and generate a two column list of device ids and names.

To understand what that code does, you need to look at how the data is rendered on the page. The device list is stored in a HTML table, with each row looking like this

<tr id="1" tabindex="-1" role="row" class="ui-widget-content jqgrow ui-row-ltr">
    <td role="gridcell" style="text-align:center;display:none;width: 34px;" aria-describedby="grid-table_cb">
        <input role="checkbox" type="checkbox" id="jqg_grid-table_1" class="cbox" name="jqg_grid-table_1">
    </td>
    <td role="gridcell" style="" class="ui-ellipsis bold" title="iPad" aria-describedby="grid-table_name">iPad</td>
    <td role="gridcell" style="display:none;" class="ui-ellipsis" title="c" aria-describedby="grid-table_status">c</td>
    <td role="gridcell" style="" class="ui-ellipsis" title="twletpb659m0ju078namuy8xnv2j0fzt1kytanfz" aria-describedby="grid-table_deviceNumber">twletpb659m0ju078namuy8xnv2j0fzt1kytanfz</td>
</tr>

Looking at the highlighted lines 6 and 9, we can see the device name and device id as the text of table cell tag. Each cell has a aria-describedby attribute to identity the type of value being stored. We can search on the values of the attributes to locate the data that we want. Going back to the javascript, look at the following lines:

var names = ["Device Name"];
$("td[aria-describedby=grid-table_name]").each(function(){
    names.push($(this).html());
});

The first line declares a Javascript array with an initial array element of “Device Name”. The next line performs a jQuery select for all of the <td/> elements that have attribute of aria-describedby with the value grid-table_name. The next part of the statement iterates over the list of matching <td/> elements and uses the jQuery html() to get the text value of the cell and add it to the array. We then can then do the same technique to get the device id and then build a list as a string and finally dump it to the browser’s console.

I also needed to the email addresses of all of our registered developers. The email addresses were not in a table, but part of a list. Each email address is wrapped inside a section element like this

<section class="col-100 ng-scope">
  <p ng-bind="::person.fullName" class="ng-binding">First Last</p>
  <a class="smaller ng-binding" 
    ng-bind="::person.email" 
    ng-href="mailto:first.last@yourcompany.com" 
    href="mailto:first.last@yourcompany.com">
    first.last@yourcompany.com
  </a>
</section>

I just needed the text part from the <a/> element. Getting the email addresses was a simpler version of the code to get the devices. I just a jQuery select on the ng-bind attribute and matched on the value “::person.email”. That ended up being a single line of code to run in the browser’s developer console

$('a[ng-bind="::person.email"]').each(function(){
  console.log($(this).text())
  });

And that’s how you can screen scrape data from a web page that doesn’t provide any support for exporting the data.

Bonus round
The aria-describedby attribute is a commonly used accessibility element used to describe the element that the tag is part of. The “aria” part of the attribute name is an acronym for Accessible Rich Internet Applications. Among other things, it was designed to allow assisted reading devices help parse a page for users with visual difficulties. It’s a good technology to use on your web pages.