Since Tact is a messaging app where users do expect to be notified of new messages reasonably fast, Tact in May 2026 switched to visible notifications delivered from CloudKit. Those do work reliably and fast, but the user experience is not as fast as it could be. CloudKit has several bugs in this area, and the bugs compound.

So this is simply an inventory of open bugs filed to Apple around the area of CloudKit, notifications, and the related developer experience, as of May 2026.

FB22867206: Notification service entitlement request page not available

Notification Service Entitlements allow iOS Notification Service Extension to suppress displaying a received notification. This would be a handy capability to have in Tact, and let us work around many of the other bugs. As a developer, you cannot do this by default, you have to apply for a special entitlement. The application page appears to be broken or nonexistent.

The page redirects to this link: https://developer.apple.com/contact/request/notification-service

Accessing the link redirects me to sign in with my Apple account. After I sign in, I am redirected to https://developer.apple.com/unauthorized/ that has text “Your account can’t access this page. There may be certain requirements to view this content. If you’re a member of a developer program, make sure your Account Holder has agreed the latest license agreement.” Screenshot of the page attached.

All my Apple Developer accounts are in good standing and have agreed to the latest license agreement.

FB22866852: CloudKit sends duplicate CKDatabaseSubscription notifications for child record changes

I am seeing duplicate CloudKit database subscription push notifications for a single logical record creation. This happens in both private and shared databases.

Tact’s CloudKit record structure is that the root CloudKit record (possibly shared with other users) is Chat, which can contain Message children, and each Message child can further contain a number of Reaction. Now, when you add a Reaction to a Message with user A, then user B sometimes receives this sequence of events:

Received remote notification
Push #2 CloudKit notification type: 4, subscription: private-database-visible-subscription
Database notification scope: private
Change fetch #3 scheduled because: push #2; scopes: private
Change fetch #3 starting
Loaded database token for private, has token: true
Stored database token for private, has token: true
Database changes for private: changed 1, deleted 0, purged 0
Fetching zone changes for 1 private zones
Loaded zone token for Chats/__defaultOwner__, has token: true
Stored zone token for Chats/__defaultOwner__, has token: true
Zone changes for private: changed 2, deleted 0
Changed records for private by type: Message: 1, Reaction: 1
Changed record IDs for private: Reaction:41AE3D75-CECF-4DE3-8623-BE0AED17A1F7, Message:4F4E8BE7-3247-4721-B673-619D8930D9E3
Upserted reaction 41AE3D75-CECF-4DE3-8623-BE0AED17A1F7
Upserted message 4F4E8BE7-3247-4721-B673-619D8930D9E3
Change fetch #3 finished

Received remote notification
Push #3 CloudKit notification type: 4, subscription: private-database-visible-subscription
Database notification scope: private
Change fetch #4 scheduled because: push #3; scopes: private
Change fetch #4 starting
Loaded database token for private, has token: true
Stored database token for private, has token: true
Database changes for private: changed 1, deleted 0, purged 0
Fetching zone changes for 1 private zones
Loaded zone token for Chats/__defaultOwner__, has token: true
Stored zone token for Chats/__defaultOwner__, has token: true
Zone changes for private: changed 0, deleted 0
Changed records for private: none
Change fetch #4 finished

When I add a reaction, I expect to receive only database change notification. Even if several records are modified because of this, they could be coalesced, with only a single database notification sent. Instead, you receive two database notifications, and must somehow deal with that.

These log lines come from the example app that I created to confirm this bug. The bug cleanly reproduces for me as of May 2026 with platform (OS and tools) version 26.5.

FB22867218: Unable to create CKQuerySubscription in shared CloudKit database

Observed: I cannot create CKQuerySubscription in shared CloudKit database. In my app, I work with record structures that can either be owned by myself, living in private CloudKit database, or shared to me by others, living in shared CloudKit database. In my app, these records should behave mostly the same, and I would like to receive similar notifications about them.

Expected: I should be able to create CKQuerySubscription in shared database exactly as I do in private database.

Not having CKQuerySubscription available in shared database means I must rely on database change notifications in shared database to generate notifications, which are less accurate and sometimes generate me redundant notifications (FB22866852). I also cannot suppress the redundant notifications on the UI side (FB22867206).

Being able to create CKQuerySubscriptions in shared CKDatabase would alleviate some of the other bugs. I doubt this will ever happen, because CloudKit seems to be architected in a way where this is technically difficult. So I should be able to work around this, but the other bugs show why this also is problematic today.

FB22867222: Cannot manage subscriptions or do anything else in private CloudKit database in CloudKit console when developer account has Advanced Data Protection enabled

How to reproduce:

• Enable Advanced Data Protection on the Apple account that you use for Apple development.
• Go to CloudKit Console
• Try to access records in a private or shared database.

Expected: you can access records.

Observed: private and shared database display an error, “Failed to access iCloud data. Please sign in again.” (screenshot attached)

Signing in again has no effect and I end up in the same state.

I have icloud.com access enabled on my iCloud account.

FB22867235: Creating a production CloudKit subscription requires deploying schema changes with no visible indication to developer

How to reproduce:

• Create a new subscription in debug version of your CloudKit app against development environment.
• Try to work with the subscription in development CloudKit environment. This works as expected.
• Deploy your app into a production environment (e.g TestFlight) and run the same code there.

Expected: subscription should work the same way as it did in deveopment environment.

Observed: you get this error:

error=CKSubscriptionError(code: 12, localizedDescription: "Error saving record subscription with id PrivateReactions-v1 to server: attempting to create a subscription in a production container", retryAfterSeconds: 0.0, errorDump: "<CKError 0xad4844390: \134"Invalid Arguments\134" (12/2006); server message = \134"attempting to create a subscription in a production container\134"; op = 81412ECB4A4DD294; uuid = DA9E1B02-ADA6-4E9D-A5EA-50162F1311EF; container ID = \134"iCloud.com.justtact.Tact\134">")

Known workaround: deploy CloudKit schema from development to production. The schema UI in CloudKit Console does not indicate that there are subscription changes or any other changes to the schema, but it seems that deployment is actually needed to remove this error in production environment. After I deploy schema changes and wait for a little while, the error disappears in production CloudKit environment.

Related thread in Apple Developer Forums

Tact operates a small server that delivers Apple push notifications to your devices. Tact apps send messages to this server. To know who to deliver notifications to, we store membership info of chats. All Tact users are identified with opaque salted+hashed iCloud user identifiers (long string of numbers and letters, not your e-mail, phone or any other private info). We have no way of connecting these identifiers to actual users. The notification server does not store any message content: it only transforms the content in memory, and then sends it to Apple push notification servers.

TL;DR: This paragraph, and the mentioned server whose internal name was tact-notifier, are now gone. The privacy of Tact is improved because we no longer have to move any message content through our systems, and we no longer have to know who to deliver notifications to. There is no longer any server component to Tact. Everything travels directly from your device to iCloud, and from iCloud to your device.

Let’s take a step back and look at how CloudKit notifications work, how we use them in Tact, and what role tact-notifier played in all of this.

Notifications in messaging apps

How does a messaging app know whether it should deliver a notification to you when someone sends a message?

What a silly question, you might say. When you are in a chat with someone, and they post something, you expect to see a notification on your phone.

… except if you have muted the chat.

… but maybe you have still chosen to receive notifications when you are mentioned, even if the chat is muted.

… but you already read the message on another device, so you expect no notification.

It gets more complicated than you initially thought. Here is a well-known diagram that Slack shared a few years ago, about how Slack decides whether to deliver a notification to a user.

This is meant just as an illustration, and does not claim to be a complete or accurate description of how notifications are delivered in Slack. And this was several years ago, and Slack has evolved since then, so this might no longer be accurate.

I haven’t made this diagram for Tact, but the essence is the same. There are many factors that go into deciding whether a notification should be delivered.

Notification capabilities of CloudKit and the Apple platform

You could say that the above was the “requirements” for a notification system from the user perspective. Let’s look at what CloudKit and the Apple platform more broadly offer to meet these requirements.

CloudKit’s interface to notifications is CKNotification and its concrete subclasses, most interestingly CKQueryNotification, CKRecordZoneNotification, and CKDatabaseNotification. You create subscriptions to these types of notifications using the matching subscription classes, CKQuerySubscription, CKRecordZoneSubscription, and CKDatabaseSubscription. Whenever there is a data change in CloudKit that matches a subscription criteria, CloudKit will post a notification to your app.

One thing you’ll notice is the discrepancy of subscription availability between CloudKit database types. CKQuerySubscription is only available for public and private databases, and CKDatabaseSubscription is only available for private and shared databases. Tact uses all three databases, and chats live in both private and shared databases. I wanted to have the same kind of interface for chats living in both shared and private databases, and the only common denominator that I can use between them is the CKDatabaseSubscription which basically tells me “something changed in this database”. CKRecordZoneSubscription is also available in both private and shared databases, but I haven’t used it in Tact.

One other Apple platform tool I will throw into the mix is UNNotificationServiceExtension. This is a tool that you can use to transform received user-visible push notifications on the device. You can imagine a system that receives a CloudKit notification, downloads the changed data from CloudKit, and uses the notification service extension API to transform the notification based on the data received from CloudKit.

The one wrench in this system is that UNNotificationServiceExtension must always return a user-visible notification. This is not desirable if you received a message in a chat that you muted, and don’t actually wish to display anything to the user. Apple sort of imagined this scenario, and it offers a way to suppress notifications in the service extension. But you must apply for an entitlement to do this. I applied months ago with Tact, and have not heard back, and I don’t think I ever will.

Client vs server notifications

How does a notification end up being displayed on the user’s device? Some part of the system must originate the push and generate the user-visible notification. But note that these two things do not need to be coupled. There are at least two ways to end up with a displayed notification:

1. The server runs the logic to determine whether a notification should be shown, and if so, generates and sends a visible push notification.
2. The server sends a silent push notification. The client runs the logic to determine whether a notification should be shown, and if so, locally generates and displays a visible notification to the user.

If you have even a marginally complex data model in CloudKit, there is no way to run the logic on CloudKit to determine whether to send a notification, especially in shared databases. (You may be able to do it in a private database if you can construct a CKQuery that captures all the logic.) It would be cool if there was a way to run this logic on CloudKit with something like a cloud function that you write, but this does not exist.

tact-notifier was born as a stopgap to allow us to run the notification logic on the server side. It kept track of user chat memberships and device tokens, and also user’s state (which users had muted which chats). When you sent a message with Tact, the client actually sent it to two places: CloudKit for persistence, and tact-notifier for notification delivery. tact-notifier examined the state of each user that was in the chat where the message was sent, and generated the push notifications for the users that needed them. It also ran slightly similar logic for reactions. (For messages, you want everyone but the sender to receive a notification. For reactions, you want only the sender of the message being reacted to, to receive a notification about it.)

Downsides of tact-notifier

What was so bad about tact-notifier, and why am I happy that it’s gone now? Couldn’t I keep running it?

There were several downsides to tact-notifier.

It broke the privacy promise. Tact says that we as the Tact team don’t have any access to user data, and everything moves directly between your device and iCloud. tact-notifer broke this promise, as your private data traveled through our server. Sure, we didn’t look at it, but just the fact that the data moved through our system at all, always made me uncomfortable. I am glad this is no longer the case.

It was another thing to operate and maintain. The overhead wasn’t too much, but it was still something that had to be done. It feels much better now to have a system that literally has no server components for Tact’s core messaging functions which we would have to maintain.

Correctness was not guaranteed. Chat membership info in tact-notifier was relying on API calls consistently arriving from clients. This was mostly the case, but there was no hard technical guarantee that tact-notifier had the same view of chat memberships as CloudKit and the clients did. There is no CloudKit server API to private or shared CloudKit databases, so tact-notifier can’t consult the authoritative source of truth.

The API was insecure. Let me say upfront that there were no privacy risks and ways to exfiltrate info from tact-notifier, since all the client-facing API calls were write-only and didn’t return any information. But the API was still insecure and unauthenticated. If you had sniffed the protocol, you would have figured out how to send rogue notifications as yourself or other people to chats where you are a member, of even to other chats if you got lucky and somehow guessed the chat UUIDs. The only reason why we had no security incidents around this is that Tact is a small and obscure app and no one bothered to target it. This was not sustainable in the long run, and I always thought one of two things will need to happen: either we shut down the server (as we now did), or we properly secure the API.

How would you secure an API like this? One way would be to authenticate all the user requests with signatures. The system could generate private keys for all users which are kept in their private CloudKit databases, and publish the corresponding public key to public CloudKit database, which tact-notifier could retrieve and use to verify the requests signed by the users. This would have been additional complexity in the system, and I’m glad I didn’t have to implement it.

A new direction with silent notifications

So this is where I was earlier in 2024. I needed to run the logic somewhere either on client or server side for delivering the notifications. There is no way to run it in CloudKit. I could have run it in the client with a notification service extension, but I didn’t have the entitlement to mute unwanted notifications. tact-notifier was insecure and unsustainable. What to do?

I had a lab with friendly Apple CloudKit engineers during WWDC 2024 where I discussed all this, and asked what they would recommend. They suggested an avenue that I sort of knew about, but had previously dismissed: why not send silent notifications from CloudKit, and generate the user-visible notifications on the client if needed, based on the user’s state that is fully available on the client?

Architecturally, this is neat and fits into the Tact vision, where things travel directly between the user’s device and iCloud. Having full state available on the client means that the app can reliably compute and present the notifications.

The one wart in this approach is that Apple does not guarantee reliable silent background notification delivery. Their documentation has this quite ominous warning:

The system treats background notifications as low priority: you can use them to refresh your app’s content, but the system doesn’t guarantee their delivery. In addition, the system may throttle the delivery of background notifications if the total number becomes excessive. The number of background notifications allowed by the system depends on current conditions, but don’t try to send more than two or three per hour.

I still coded it up and we tested it in Tact for a while. Not least because the Apple engineers in the lab said that several Apple apps use the same approach. (But they asked me to not quote them on that, so I won’t, this is a loose paraphrase.)

The good news is that background silent notification delivery is one of the rare areas where Apple is underpromising and overdelivering. Background silent notifications arrive quite fast and reliably and aren’t really throttled to just a few per hour. The delivery is not 100%, but it’s still quite reliable.

So, this is what Tact does now. It subscribes to silent CloudKit database change notifications. When those arrive, it downloads new data from CloudKit for the given database, and when there are new messages or reactions for the current user that match the delivery criteria (e.g the chat is not muted), the client generates local visible notifications.

Shutting down tact-notifier

Here is a bonus video to end this post. How I shut down tact-notifier.

You see this warning in your console when you run Canopy tests or use Canopy’s MockCKRecord type in your own code, with Canopy versions earlier than 0.5.0.

Canopy 0.5.0 is a fix to this warning, and you no longer see it in this version.

To fix this warning, I re-thought a significant part of Canopy’s types.

TL;DR

All Canopy request results now return CloudKit record values as CanopyResultRecord.

You should access the data via CanopyResultRecordType protocol, which is identical to CloudKit API in terms of getting data. There are no setters, it’s a read-only view into the record.

For mocking CloudKit data with Canopy, you should now use MockReplayingContainer and MockReplayingDatabase in most of your tests. (This is not strictly related to the previous points, but emerged as a natural ergonomic improvement.)

The rest of this post is a deeper look.

The problem

CKRecord is a central part of CloudKit API. You use CKRecord both to store values into CloudKit, and to retrieve values out of CloudKit.

There are some fields of CKRecord that are only set by the CloudKit system on the server side, and that the client should treat as immutable: creationDate, creatorUserRecordID, modificationDate, and lastModifiedUserRecordID. For production use, this is fine. In a system like Canopy, though, there needs to be a way to override values of these fields for mocking and testing purposes.

Previously, Canopy offered MockCKRecord that essentially did this for all these fields:

public class MockCKRecord: CKRecord {
  public static let testingCreatorUserRecordNameKey = "testingCreatorUserRecordNameKey"
  
  override public var creatorUserRecordID: CKRecord.ID? {
    guard let testing = self[MockCKRecord.testingCreatorUserRecordNameKey] as? String else {
      return nil
    }

    return CKRecord.ID(recordName: testing)
  }
}

This worked fine, even if it was a little bit hacky. But in the Swift Concurrency world with Sendable, this no longer flies, and produces the above error.

It’s likely that CloudKit won’t actually change any time soon, and the above MockCKRecord would remain working. At least the Apple platform of 2024 (iOS 18 and macOS 15) is not this “near future release.”

Nevertheless, this all felt iffy, and triggered an introspection of what all the related Canopy API-s should really mean and be, and how to best express the desired behavior in the new Swift world of 2024.

CKRecord’s role in Canopy, and designing the Swift API with protocols

The above MockCKRecord was a somewhat half-hearted approach to mocking, which now hit a Swift language design limit with Sendable.

There are two avenues for mocking that I could now have taken. First, since CKRecord is an Objective-C object, I could perhaps have mocked it with something like OCMock or anything else that hacks the ObjC runtime to provide the desired mock functionality. Objective-C remains a supported language and this probably would have worked with less changes to Canopy that the other approach I took. But, this didn’t feel future-proof and good.

The other approach, which I ended up taking, is thinking in terms of protocol-oriented design that Swift encourages, and ask the question: what is the role, the job, of CKRecord in CloudKit and Canopy? And what are some other ways to represent these jobs in API-s, other than using CKRecord directly?

CKRecord is a feature-rich object that has two distinct jobs in CloudKit that mirror each other:

1. represent the local state of a record to CloudKit (posting records to CloudKit)
2. represent the CloudKit state of a record to local state (receiving records from CloudKit)

These are related but distinct. Since the problem with mocking only occurs with 2 but not 1 (indeed, in this version of Canopy there are no changes to the API parameters that upload state to CloudKit, which continue to use CKRecord), we can just constrain ourselves to 2 and ask, are there other ways to represent an object’s state, besides using CKRecord directly?

CKRecord is a mutable object. We can simplify the problem by stating that for transferring record representations from CloudKit into local state, we are only interested in immutable read-only representation that should be short-lived and get transformed from CKRecord-like-state into local storage state without mutations to the CKRecord-like-state.

Canopy 0.5.0 captures this idea in the CanopyResultRecordType protocol:

public protocol CanopyRecordValueGetting {
  subscript(_ key: String) -> CKRecordValueProtocol? { get }
}

protocol CanopyResultRecordType: CanopyRecordValueGetting {
  var encryptedValues: CanopyRecordValueGetting { get }
  var recordID: CKRecord.ID { get }
  var recordType: CKRecord.RecordType { get }
  var creationDate: Date? { get }
  var creatorUserRecordID: CKRecord.ID? { get }
  var modificationDate: Date? { get }
  var lastModifiedUserRecordID: CKRecord.ID? { get }
  var recordChangeTag: String? { get }
  var parent: CKRecord.Reference? { get }
  var share: CKRecord.Reference? { get }
}

This represents the entirety of CKRecord API (at least all the parts that Tact needs) constrained to the idea of getting the record value, but not being able to set it. Whenever you see CanopyResultRecord in Canopy API-s, you should most of the time think CanopyResultRecordType and just work with this protocol.

The API of CanopyResultRecordType is identical to CKRecord for the getters, and that’s intentional, to ease the call site migration. For most use, there shouldn’t be any migration required at all in the feature code that just reads CKRecord values and transforms them into local state.

CanopyResultRecord

CanopyResultRecord is a new type that represents a record as read-only. Internally, it’s powered by one of two data sources: either a CKRecord (received from CloudKit or created locally, both are fine), or another new type, MockCanopyResultRecord that’s a simple value type where you can set all the above fields on creation, which is what you often want to do in unit tests.

CanopyResultRecord is both Codable and Sendable, meaning that you can easily transfer it across isolation domains, and archive and unarchive single objects and their collections to shuffle around representations of entire mock databases and containers.

New mock database and container types

When working on the above, it occurred to me to simplify approach to testing and how you set up the test data. Previously, the way to do this in Canopy was with ReplayingMockCKContainer and ReplayingMockCKDatabase that provided mocked representations CKContainer and CKDatabase state. You initialized them with CloudKit API responses, and the data flowed through the entire Canopy logic and got transformed into a Canopy API response.

Doing this remains supported, and is an important tool for testing Canopy’s own functionality and logic. I realized, though, that if you are working on application code, it would be easier to mock Canopy API responses directly. You can now do this with the new types ReplayingMockContainer and ReplayingMockDatabase.

This means that when properly coded against Canopy, your feature code may be receiving real or simulated CloudKit responses from three places:

• From real CloudKit via CKDatabase and CKContainer
• From simulated CKDatabase and CKContainer via ReplayingMockCKDatabase and ReplayingMockCKContainer
• From simulated Canopy API via the new ReplayingMockDatabase and ReplayingMockContainer

The nice thing is that your feature code with Canopy doesn’t care which of these three it is, and continues to work all the same, with everything nicely testable.

The diagram below summarizes the possible paths that a database request may take in Canopy. Note how only the live use hits iCloud. The other two remain entirely local to the device, and in both cases, the dependencies are controlled.

Extras

Above were the most important changes in Canopy 0.5.0. I ran into a few more things that I’ll document here.

CKShare

Canopy record mocking has currently no explicit support for CKShare. You can’t mock CKShare the same way that you mock CKRecord. Internally, Tact subclasses CKShare the same way that Canopy subclassed CKRecord, and it yields a similar warning as you see in the beginning of this post.

CKShare is a strange class in that subclasses CKRecord, but they actually have very little in common, and are doing very different jobs in CloudKit. Inheriting from a common base class or implementing a protocol for the base metadata properties would make more sense, but the CloudKit API design is stable and unlikely to change.

Canopy 0.5.0 provides a small escape hatch for working with CKShares. CanopyResultRecord has this getter: public var asCKShare: CKShare? So if you initialize CanopyResultRecord with a real CKShare, you can get it back this way.

Proper mocking of CKShare may get added in a future Canopy version.

Using DocC across multiple SPM targets

The Canopy documentation site is currently half-broken about the types documentation, because Canopy types are spread across multiple Swift Package Manager targets, and DocC does not currently support generating documentation out of box for such case. This thread in Swift Forums links to ongoing work about this.

There is a more hacky way to do this, but I decided this is currently not worth the maintenance effort for Canopy, so I am waiting for this feature in the official toolchain.

Inline documentation for Canopy in Xcode works fine and is not affected by this. It’s only the DocC documentation site suffering from this shortcoming.

Xcode swift-frontend compiler hallucinating errors in GitHub Actions

I was using GitHub Actions with Xcode runner for Canopy’s CI, to build and test the code. This worked fine for a while, but with Canopy 0.5.0, Xcode in GitHub Actions, more specifically swift-frontend, started hallucinating a nonsensical error. GitHub Actions community thread. Swift bug. Link to a failed job.

The code builds and runs fine for me locally with the same Xcode version (15.4), the code is warning-free and error-free.

I suspect that it is something on the GitHub Actions setup side, because I moved Canopy CI to CircleCI, where this error does not occur. CircleCI has a nice open source offer which I am happy to use and recommend now.

This is one of those posts, exploring how querying based on CKReference works, and what are the limits and behaviors of that.

Let’s start from the basic schema, and work our way up from that.

The Tact schema

Tact has the following schema for its chats, messages, and reactions.

It’s as basic as you’d expect: a very simple tree structure. Chat is the top-level object that provides all the context and permissions. A chat can have zero or many messages, and each message belongs in exactly one chat. A message can have zero or many reactions, and each reaction belongs to exactly one message.

In CloudKit terms, these relationships are expressed with the parent property of CKRecord. Message records have their parent set to the chat record, and reaction records have their parent set to the message record. Parent relationships establish the permissions, among other things: in Tact, everyone who can see a chat, can also see all the messages and reactions in that chat.

Here is one important bit: you can’t do queries by parent property. I couldn’t immediately find the authoritative source for this in the SDK documentation, but the recommendation in Apple forums and elsewhere has always been to add your own CKReference fields if you want to query by parent record. So the pattern I use in Tact is that all child records have a relatedRecords field which is a set of CKReferences. A message’s relatedRecords contains reference to the chat, and a reaction’s relatedRecords contains refererences to the message and chat. So I can do queries like “give me all messages for this chat”, “give me all reactions for this message”, and so on.

Designing “Load more” queries

I recently updated the “Load more” functionality in Tact, and encountered some unexpected behaviors that this post documents.

Tact can host chats that literally continue for years and have lots of messages, reactions, pictures, videos, and other content. If you install Tact on a new device, it only loads recent content for each chat by default, but you can “Load more” yourself if you want to go back in time. The user experience of “Load more” may vary, and will improve in future Tact versions, but from the system point of view, it’s always the same: you want to load some more messages and reactions in a given chat.

Here’s how I set up the query to load messages.

let chatRecordId = task.model.chat.cloudKitRecordID
let referenceToMatch = CKRecord.Reference(recordID: chatRecordId, action: .none)
let date = task.model.earliestMessage?.date ?? Date()
let predicate = NSPredicate(format: "creationDate < %@ AND relatedRecords CONTAINS %@", date as NSDate, referenceToMatch)
let query = CKQuery(tactRecordType: .message, predicate: predicate)
query.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
let result = await databaseAPI.queryRecords(
  with: query,
  in: task.model.chat.cloudKitRecordZoneID,
  resultsLimit: 100,
  qualityOfService: .userInitiated
)

In a human language this is: query for messages for a given chat whose creation date is earlier than the date of the earliest known message in the chat (or current date if there are no earlier messages), order them by creation date in descending order, and retrieve 100 of those messages. (Why 100? I’ll get back to that.)

(Side note: all this assumes that you have relevant indexes set up on the CloudKit side, which I will not cover in this post.)

Now, how to query reactions? You might think of doing the same and querying them based on the creation date and chat. The schema affords that. But consider that reactions might get created at a different time than the messages they refer to. When you just query for reactions based on time, you may get reactions which refer to messages that you don’t have locally, and you may miss reactions for messages that you do have, if the reaction creation time falls outside of the time window.

So, for consistency, it makes sense to me that I first get the messages, and then get the reactions for exactly those messages. Here is how to set up such a query, if I have previously obtained a bunch of message records:

let messageIdReferences = messageRecords.map { CKRecord.Reference(recordID: $0.recordID, action: .none) }
let predicate = NSPredicate(format: "ANY %@ in relatedRecords", messageIdReferences)
let query = CKQuery(tactRecordType: .reaction, predicate: predicate)
let result = await databaseAPI.queryRecords(
  with: query,
  in: recordZoneID,
  qualityOfService: .userInitiated
)

The key bit is the predicate for the reference set. I got the idea for setting up this way from this StackOverflow post. The official CloudKit documentation does have examples for querying based on references, but not reference sets and more complex scenarios.

Now, if you just attempt to run all this, what do you think will happen? Will it all work out of the box?

Here’s where the interesting part of this post begins. 😀

How reference-based queries behave with various inputs

Here’s what will happen if you try to run the above queries with various inputs.

First, let’s say that you don’t want to limit the number of message ID inputs to your reactions query. You just say, let’s YOLO, and retrieve all messages, and all reactions for all of those messages right away. The message side of this works fine: if you have many messages, CloudKit will return a page of messages, and a cursor to retrieve the remaining ones, and you rinse and repeat this until there is no more cursor. And it’s all reasonably fast. You can retrieve thousands of messages distributed across several pages in just seconds.

Now, if I attempt to grab all those thousands of messages and run the above reactions query with them, here’s what I get:

Error querying chat reactions: CanopyTypes.CKRecordError(code: 27, localizedDescription: "Query filter exceeds the limit of values: 250 for container \'iCloud.com.justtact.Tact", retryAfterSeconds: 0.0, errorDump: "<CKError 0x600002ba6b50: \"Limit Exceeded\" (27/2023); server message = \"Query filter exceeds the limit of values: 250 for container \'iCloud.com.justtact.Tact\"; op = DAD629CFCE8A1C85; uuid = 60D6BC5A-8ED7-489D-9C75-08A8433E3C1C; container ID = \"iCloud.com.justtact.Tact\">", batchErrors: [:])

I have wrapped the CloudKit error in some types, but you see the CloudKit error right there. CKError code 27 is indeed CKErrorLimitExceeded.

Fair enough. This tells me that the filtering condition in the clause ANY %@ in relatedRecords can’t contain more than 250 objects. So, let’s limit it, leave some headroom, and run it with 200 records instead.

If I run the reactions query with 200 message records in the filtering condition, here’s what I get:

Error querying chat reactions: CanopyTypes.CKRecordError(code: 15, localizedDescription: "Request failed with http status code 500", retryAfterSeconds: 0.0, errorDump: "<CKError 0x6000032f6bb0: \"Server Rejected Request\" (15/2001); \"Request failed with http status code 500\"; uuid = 1A749162-6811-45E4-87E7-2A61F12A50D2>", batchErrors: [:])

Even though the input contains only 200 objects, I get this error. I speculate that even though the input amount is below the nominal stated limit, some relationship query to construct the results causes some internal limit overflow on the CloudKit side, and it doesn’t handle this situation well.

Over several days, I got different CKErrors in the same situation when doing this same query. Sometimes I got code 6, serviceUnavailable, and interestingly, it also had rate limiting set (retryAfterSeconds was >0). When I tried to YOLO and did another query without honoring the rate limit, I then expectedly got CKError code 7, requestRateLimited. Sometimes I also got CKError code 12, invalidArguments.

The way to recover from all of these is to split the input set into two, and re-run the query with those two smaller batches (respecting the retryAfterSeconds parameter), and merge the results. I built some code to do that, and it appears to work fine. With Canopy, it’s easy to build deterministic fast tests for it to assert it behaves correctly in all cases, but I didn’t move this code (to re-run the CKReference queries with smaller batches if the initial batch is too big and returns an error) itself to Canopy yet.

I mentioned that the optimal input set to the filter is 100 message records because when I was testing one day with batches of 200, I always got serviceUnavailable with rate limiting set to something like 20 seconds. Re-running the query immediately is fine, but running it with a large rate limit is bad user experience, when the user has to wait for results for a long time. But when I ran it with 100 records, everything worked as expected, and I never got the rate limit, and that’s why I am keeping it at 100 right now. It’s possible that CloudKit varies its server-side behavior and the errors it returns based on its usage load and possibly other parameters that I don’t know and can’t control.

Conclusion

You can do advanced CloudKit queries based on CKReference set fields that tie multiple levels of records together, but be aware of the system limitations and quirks. Often, the error is recoverable by modifying your input parameters into smaller batches, and re-running your query with multiple smaller batches. In some cases, you may get rate limited, and choosing a smaller default input size may help you avoid the rate limiting for best user experience.

Today, we are sharing this with you in the form of Canopy library. Canopy is available on GitHub.

Canopy helps you write better, more testable CloudKit apps. The project has three parts.

Libraries

Libraries provide the main Canopy functionality and value. Canopy is the main library, CanopyTestTools helps you build tests, and CanopyTypes provides some shared types used by both.

Canopy source is available at github.com/Tact/Canopy.

Documentation

The Canopy documentation site at https://canopy-docs.justtact.com has documentation for the libraries, as well as information about the library motivation and some ideas and best practices about using CloudKit. The documentation is generated by DocC from this repository, and can also be used inline in Xcode.

Some highlights from documentation:

Canopy motivation and scope

Testable CloudKit apps with Canopy

iCloud Advanced Data Protection

Example app

The Thoughts example app showcases using Canopy in a real app, and demonstrates some best practices for modern multi-platform, multi-window app development.

More info about Thoughts example app

Let’s talk about the other two.

Security Keys for Apple ID do not have a direct impact on Tact and its data, but are indirectly a great way to improve the security of your user account. Since Tact uses your iCloud account which in turn uses your Apple ID, do consider using a hardware security key to further secure yourself.

Advanced Data Protection (ADP) for iCloud is the most intriguing of the three, and the rest of this post will discuss how it can improve the security of your data in Tact and other CloudKit-based apps.

TL;DR for Tact: your Tact private chats will be end-to-end encrypted if all chat members have enabled Advanced Data Protection on their accounts.

TL;DR for any CloudKit app: your records in iCloud will be end-to-end encrypted if certain conditions are met. You have no way to verify some of the conditions on your end.

iCloud security basics

The security of third-party app data in CloudKit was previously a bit of a mystery. There were a number of platform security documents by Apple, but there wasn’t an easily digestible policy or guide.

With this data protection announcement, Apple has freshly published a number of documents that discuss iCloud security, including that of third-party CloudKit apps, in greater detail:

• iCloud security overview
• iCloud data security overview
• Advanced Data Protection for iCloud

The first one pretty clearly outlines that all data in iCloud, including both Apple and third-party apps, now falls under one of two policies.

• Standard data protection (the default setting): The user’s iCloud data is encrypted, the encryption keys are secured in Apple data centers, and Apple can assist with data and account recovery.
• Advanced Data Protection for iCloud: An optional setting that offers Apple’s highest level of cloud data security. If a user chooses to turn on Advanced Data Protection, their trusted devices retain sole access to the encryption keys for the majority of their iCloud data, thereby protecting it using end-to-end encryption.

In a nutshell, data in both cases is encrypted both in transit and at rest in the iCloud data centers. In the first case, Apple has access to the keys. In the second case, they don’t, and data is end-to-end encrypted if certain conditions are met.

Storing encrypted data in CloudKit

If you use CloudKit to store your app data in iCloud, you need to look at the encryptedValues API in CKRecord. This is your single entry point to storing and retrieving the encrypted values that could be end-to-end encrypted with Advanced Data Protection.

There’s three categories of data to think about.

• CKAsset records are already always encrypted.
• CKReference fields are never encrypted, since the server needs access to them to identify relations between records.
• For all other data types, they are encrypted as long as you use the encryptedValues API to work with them, instead of setting them directly on your CKRecord.

You lose certain features with encryption. As the documentation says:

CloudKit doesn’t support indexes on encrypted fields. Don’t include encrypted fields in your predicate or sort descriptors when fetching records with CKQuery and CKQueryOperation.

This is pretty obvious, but worth mentioning. Since the server no longer has access to the field contents, you obviously cannot query, search or sort by them.

Sharing

Sharing has impact on ADP and end-to-end encryption. The data security overview says:

iWork collaboration, the Shared Albums feature in Photos, and sharing content with “anyone with the link,” do not support Advanced Data Protection. When you use these features, the encryption keys for the shared content are securely uploaded to Apple data centers so that iCloud can facilitate real-time collaboration or web sharing.

It’s not clear what “anyone with the link” means for CloudKit apps and CKShare. I assume it means the publicPermission of a CKShare being something other than none: that is, if anyone can access the shared record by just opening its shared URL, even if it is only for reading purposes.

When does ADP and end-to-end encryption apply to values of a CloudKit record?

Let’s now put all the pieces together and ask: when is the data in a given CKRecord end-to-end encrypted? As far as I can tell, these three conditions must be met.

1. You use the encryptedValues API and CKAsset to store the data that you want to protect.
2. If the record belongs in a shared record hierarchy, publicPermission on the CKShare that governs the share is none.
3. Current user, and in case of shared record also all other users, have ADP enabled on their iCloud account.

As a developer, you have control over conditions 1 and 2. Here’s the thing though: you have no way to tell at runtime in your app if condition 3 is met. There is no way for you to know if the current user or other CKShare members have ADP enabled on their account or not.

If you had a way to verify condition 3, you could display a nice badge or something in your app next to the relevant CKRecord, to indicate that all the conditions for end-to-end encryption have been met, and the user can assume this data to be end-to-end encrypted. But there is no way to do this, and doesn’t sound like there will be any time soon.

I was obviously interested in doing this for Tact, so I asked in a recent Ask Apple session about this. A helpful Apple person told me in pretty clear terms:

Apple does not provide API for [checking whether a given user has ADP enabled] because we don’t want to expose users’ account choices to other users.

This makes perfect sense. I’m all for not exposing users’ account choices. At the same time, I think that it’s a valid need to provide assurance to users about whether something is end-to-end encrypted, because it may inform their choice about the kind of info that they want to contribute. These two goals (protect users’ account choices, and inform all participants about whether ADP applies to a given CKRecord) are currently in conflict, and it’s clear which one Apple has chosen. I (and I suppose also Apple) can’t imagine a system design that would satisfy both goals. So, that’s just how it is for now.

Takeaway

People interested in Tact sometimes ask me about end-to-end encryption. Until now, I had to say Tact just doesn’t have any. Now, I can say that it’s there if the above conditions are met.

Implementing any security protocol correctly, including end-to-end encryption, is hard. I have infinitely more faith in Apple’s than my own ability to do it well. I hope that they will follow up with independent security analyses and audits that’s common industry practice for these kinds of systems, to provide users and developers extra assurance in their implementation.

As a developer, since you don’t have access to users’ account choices, you have no way to definitively inform your users about whether ADP applies to the data they store in your system. The best you can do today is complete your part of the equation by using encryptedValues and CKAssets, and educating your users about enabling ADP on their account if they choose to do so.

From a technical perspective, Tact chats are nothing more than shared CloudKit records, with some extra ceremony that’s not relevant for this post. For each chat, there is a root Chat record, with a CKShare attached to it that determines all the permissions and participants of the chat. CloudKit is somewhat underdocumented and esoteric, and I have done a lot of trial-and-error experimentation to find out how it works. Here’s some of that written up for my future self.

Private chats

All Tact chats, whether 1:1 on group, start out as private chats. We create a share for each chat.

let share = CKShare(rootRecord: chatRecord)
share[CKShare.SystemFieldKey.title] = chat.name
share.publicPermission = .none
// Store the share and chat records back to iCloud

The chat owner can then add and remove participants to the chat. Here’s an outline how the owner adds a participant.

let lookupInfo = CKUserIdentity.LookupInfo(userRecordID: userRecordID)
let fetchParticipantsOperation = CKFetchShareParticipantsOperation(userIdentityLookupInfos: [lookupInfo])
// Run the operation and get a shareParticipant result
shareParticipant.permission = .readWrite
share.addParticipant(shareParticipant)
// Store the share back to iCloud

The above adds a participant to the share whose acceptanceStatus is pending. The pending participant shows up only for the owner. Other members of the share do not see the participant at this point.

To join the share, the added person needs to receive an invitation to the share, open the associated CKShare URL, and go through the joining process with system confirmation on their side. After they do that, their acceptanceStatus changes to accepted, and they show up for other members of the share as well. In other words, after a member joins a share, their info now appears in CKShare.participants for all members of the share. The role if regular members of a share is set to privateUser, while owner has of course the owner role.

There are two ways for someone to leave a share/chat. Either the owner removes them with the removeParticipant API, or they leave themselves by deleting the CKShare record. Note that it doesn’t make sense for the owner to leave a share (or chat in Tact): all shared CloudKit records must have an active owner. The only thing that the owner can do to “leave” the chat is to delete the whole record (chat).

Publicly joinable chats

First, a note about terms. We currently avoid the term “public chat” in Tact because even joinable chats are still owned by one owner who has ultimate authority over it, and the chat remains in the owner’s private database, counts against their iCloud quota, the content is strictly visible only to chat members, they are the ultimate authority about participation etc. We may choose to have another kind of “public chat” in Tact in the future, which lives in the public iCloud database with all the implications that come with it. So far, we haven’t done this. Hence, we talk about “joinable” chats in Tact.

You can make a chat joinable in Tact by “publishing a link” for it. Technically, this changes the CKShare permission.

share.publicPermission = .readWrite
// Store the share back to iCloud

Anyone with an iCloud account who gets access to the CKShare URL can open it and join the chat. When they do that, they become a CKShare participant with acceptanceStatus = accepted and role = publicUser.

In Tact, I have chosen to put an level of indirection on top of CKShare URLs, so you never see or use those directly. You see “group links”, but those map back to a CKShare URL in the end, so this is just an implementation detail in Tact.

A fact that’s not obvious from CloudKit documentation: after you change a share’s publicPermission to readWrite, the owner can no longer add participants with addParticipant API. Doing so results in an internal consistency exception. The only way to add participants to a public share is if they themselves join the share using the URL.

Even though the owner cannot add new participants, they can still remove participants with the removeParticipant API as expected. In a social network like Tact, this provides a level of control to the owners, as they can remove any unwanted members, even if they joined without the owner’s explicit approval.

What happens to the previous share participants with role = privateUser when you make a CKShare publicly joinable (change its publicPermission to .readWrite)? The documentation is not obvious on this, but my experiments show that the previous set of privateUser participants is preserved when you make a share joinable. Thus, shares that were previously private, the owner added some members, and then made it joinable, will have a mix of privateUser and publicUser participants. The owner cannot add new privateUser participants as noted above, but the previously existing members are preserved. This is good for Tact, as it means if you have a private group chat and decide to make it joinable, there is no disruption to current members.

What happens when you change a share’s publicPermission from .readWrite back to .none? The documentation is pretty clear on this:

CloudKit removes all participants if the new permission is none.

Indeed, this is what happens if you do it. All other participants are removed and only the owner remains as a participant on the share. You could say it’s somewhat counter-intuitive - shouldn’t the previous set of privateUser participants be preserved? Perhaps, but this is just how the system works, and for now we must accept it as a fact.

This is the reason why you cannot go back from a joinable to private chat in Tact: all members would be removed, and it would be counter-intuitive. If you make a chat joinable, it will remain joinable.

We might implement different ways of controlling a joinable chat’s access: perhaps you could change the joining link to a different one and invalidate the previous one, or other solutions like this. Let us know if you ever run into this in Tact, and we can further develop this area.

This post is about two changes in what you need to run Tact, and how you access the macOS public beta.

Tact now requires iOS 15 and macOS 12 Monterey

We say in our principles that we may drop support for older devices and operating systems. In practice, Tact will for the time being have “one version back” policy, supporting the current public OS version, and one previous version. We are looking ahead to this fall, where iOS 16 and macOS 13 Ventura will be released. We are already now upgrading the Tact system requirement with this in mind, and Tact thus now requires iOS 15 and macOS 12 Monterey.

This OS support will be in place for one year, until Apple announces their next platform versions at next year’s WWDC. We will re-evaluate our OS support then, based on Apple’s platform plans and the community response to Tact.

macOS public beta is now on TestFlight

We have always distributed the Tact public beta for iOS with TestFlight. For macOS, we offered the beta as a direct download with a nice custom updater, not least because Tact supported macOS 11 Big Sur while TestFlight for Mac, which was released in 2021, requires macOS 12 Monterey.

The custom updater and downloader we were using aren’t compatible with how Apple distributes software via TestFlight and app store, so we removed it from Tact and are moving Tact for macOS public beta to TestFlight.

To join or continue with Tact macOS public beta, join our TestFlight public beta with this link. The link is the same for our macOS and iOS public betas, and you participate in both together. The link shows you instructions to download the TestFlight app from App Store for your platform, where you can then install Tact, and keep updating it there. You will get emails when we publish new builds.

The case for renaming

When you join Tact, the only thing we ask you for is a name. We don’t check or care if it’s your real name, pseudonym, alias, first name, full name, or anything else. We only require that it exists, and contains at least 1 character. That’s it. You may very well choose to be known as “A”, “X,” or “Tina” in Tact.

You may set a picture for yourself, but it’s not required. If you add one, we don’t have any rules about its contents. You can change or remove your picture at any time.

You connect with other people on Tact by opening the invitation links that they share with you. By design, we don’t currently provide any authentication for the links, nor the profiles you see as a result of opening the links. We assume that you know the person whose link you are opening, and are able to validate their authenticity yourself. Most often, it’s because you already know them in real life, and they shared a link with you in a channel that you can trust, perhaps by physically being next to you while they did that.

This process of connections works quite well. The flip side, though, is that you may end up with a list of people in Tact that makes it hard to understand who’s who. Maybe you connect with three friends called “Mike”, and you now have three Mikes in Tact. Sure, there are profile pictures, but those are optional, and kind of small in the list.

Private rename helps with this problem. You can rename the people in your list to something that is more meaningful to you. The rename action is private, so the person being renamed doesn’t know that you called them something else. The rename is in sync across all your Tact devices.

How to rename someone

Let’s say I have a friend, Alice Myers, who has their picture in Tact, but enters just “A” as their name. To rename them, all I need to do is long-press on them in Tact, and tap “Rename…”.

After you rename someone, they show up in your list and in all chats with their new name.

You may be curious what original name they used, even after you rename them. To do this, view their profile by opening the chat with them and tapping the three dots in the top right corner. You then see the name that you set for them. The small text below the name, “A” in this case, is the original name they set for themselves.

To remove the private “alias” you set for someone, just rename them again, but delete everything from their name field until it’s blank. Tact then reverts to the name that they entered for themselves.

Next steps

We’re thinking of other ways to augment how you see people in Tact, perhaps by having more private fields, like notes, where you can enter more info about why and how you know someone. If you have any thoughts on this, we’d like to hear your opinion in our discussion area.

Get the software

‍For iOS, join our public beta TestFlight.

For macOS, download directly from this link.

Share your bug reports and ideas

‍Visit the Tact public beta community site on Github.

Tell your friends

Share Tact with your friends, so you actually have someone to chat with. In the Tact app, go to “New Chat“, “New Direct Message”. You then get a Tact invitation link that you can share with your friends, even if they don’t yet have Tact.

Write us directly

‍If there’s anything you’d like to discuss that you don’t feel belongs in the public discussion area, you can always write to us at support@justtact.com.

What’s the difference between Tact private beta and public beta?

‍Two things.

First, how you obtain the Tact software. During private beta, you had to e-mail us to make the case for joining. We’d receive your e-mail, add you to our beta by hand, and manually reply to you with the download links and more information. Everyone that you’d like to chat with in Tact had to follow the same process, which is obviously not ideal.

The public beta of Tact is “self service”. You no longer need to e-mail us to ask to join. For iOS, you simply join our public beta TestFlight with this link. For macOS, you can directly download the app here. The macOS app has an auto-update facility, so it automatically prompts you to install new builds as we release them.

The second difference between public and private betas: we’re now more open about what Tact looks like and how we build it. For starters, you see what it looks like on justtact.com. We also have opened our Github community site at github.com/tact/public. On that site, we provide technical information, issue reporting, and a discussion area to help you make the most of Tact during our public beta, and help us improve it.

Didn’t you say previously you want to launch a paid product? What’s with the public beta?

‍Our initial plan was to go straight to iOS and macOS App Store with public version of Tact, which includes payment. This remains our goal and our next milestone, but we introduced the additional step of public beta for two reasons.

First, figuring out the exact product packages, implementing the subscription payment, and making sure it all works well are bits of work we just haven’t yet done. We’ve so far focused on the main product features and quality and decided to tackle the payment as our next thing, while the rest of the communication features of Tact are already usable and testable.

Secondly, the product quality is not yet fully there. Tact is entirely usable for daily communications, but there are more details we’d like to fix and polish before we can in good conscience ask for payment. So we decided to open the full Tact experience to the public in the form of public beta, and ask for your assistance in testing and polishing it.

When will you launch the public paid version of Tact?

‍When the product is ready. We aren’t under external time pressure to do so, but also, we can’t endlessly finance and bootstrap the project ourselves. We expect to have more news on this some time in 2022. ‍

How has Tact evolved in 2021? What have you accomplished during the private beta period?

‍On the product side, we have solidified the technical foundation, outlined the Tact character and experience, figured out our design system, and implemented many features, from the obvious (create 1:1 and group chats, upload large pictures and files both from the main app and iOS share extension in a stable and reliable manner, “like” everything) to the quirky and unique (emoji mood and emote).

On the team side, we’ve assembled a small close-knit team of designers and engineers who greatly enjoy working on Tact and would very much like to continue with your support and blessing. To date, we’ve operated in a bit of vacuum, which feels lonely at times. One of our goals with the public beta is to get some external feedback and validation to keep us going.