<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.2">Jekyll</generator><link href="https://zainp.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://zainp.com/" rel="alternate" type="text/html" /><updated>2026-04-01T21:21:38+00:00</updated><id>https://zainp.com/feed.xml</id><title type="html">Zain Patel</title><subtitle>Zain Patel&apos;s personal home page</subtitle><entry><title type="html">Harvest and carry-forward your ISA allowance :ear_of_rice:</title><link href="https://zainp.com/harvest-your-isa-allowance/" rel="alternate" type="text/html" title="Harvest and carry-forward your ISA allowance :ear_of_rice:" /><published>2022-10-05T00:30:00+00:00</published><updated>2022-10-05T00:30:00+00:00</updated><id>https://zainp.com/harvest-your-isa-allowance</id><content type="html" xml:base="https://zainp.com/harvest-your-isa-allowance/"><![CDATA[<p>The old maxim that your ISA allowance is ‘use-it-or-lose-it’ in a given tax year and that there is nothing more sacred than ensuring you’ve maxed out your ISA is true, <em>but</em> like with all things in the UK tax system, you can work around this and <em>effectively</em> carry forward your ISA allowance if you aren’t able to fill it in for a given year, for whatever reason.</p>

<h2 id="pre-requisites">Pre-requisites</h2>

<ul>
  <li>You need to have sufficient liquid capital on hand, or at least the ability to borrow significant amounts of capital for at least a couple of days</li>
  <li>You are eligible and able to subscribe capital to an ISA for each tax year in which you want to carry the ISA allowance for that tax year forward</li>
  <li>You have a <a href="https://www.gov.uk/guidance/manage-isa-subscriptions-for-your-investors#f-isa">flexible ISA</a></li>
</ul>

<h2 id="how-does-this-work">How does this work?</h2>

<p>In the general case of ISAs, the limit on the amount you can put into an ISA in a given year is £20k. That’s pretty simple - the ISA doesn’t care about how much that £20k grows to or how much you take out. If you put in £20k and realise you’ve made a massive mistake and take it back out the next day, your allowance has been used up for the year. You can’t put any more money into an ISA.</p>

<p>A flexible ISA works slightly differently, and more favourably (to the taxpayer, that is). What is a flexible ISA, you ask? In the words of HMRC:</p>

<blockquote>
  <p>A flexible ISA is an ISA whose terms and conditions allow the investor to replace, in whole or in part, cash they have withdrawn, without the replacement counting towards their annual subscription limit.</p>
</blockquote>

<p>As an example, in the same situation above: where you put in £20k and then instantly remove it again, your remaining allowance remains at £20k - so you can put money back into your ISA with the very important caveat that you put the money back in the <strong>same tax year</strong> as when you took the money out.</p>

<p>Okay, you understand what a flexible ISA is. How do we actually use this to carry forward your allowance? Let’s look at the situation of a 5 year journey where you don’t want your capital sitting inside an ISA for whatever reason, but you want to reserve the option of dumping £100k at the end of the 5 years. In the typical situation, this is what we would do:</p>

<ul>
  <li>2018/19: £0k into an ISA, £20k outside</li>
  <li>2019/20: £0k into an ISA, £40k outside</li>
  <li>2020/21: £0k into an ISA, £60k outside</li>
  <li>2021/22: £0k into an ISA, £80k outside</li>
  <li>2022/23: £20k into an ISA, £80k outside</li>
</ul>

<p>you’d be stuck with £80k left outside the ISA. However, if you’re happy doing some bank transfers across tax years, what you can do is this:</p>

<p>On 5 April 2019 (the end of the tax year), you <em>temporarily</em> put £20k into your ISA (remember the requirement of having sufficient capital) from a source of capital you have (e.g savings or magin). On the next day, 6 April 2020 (the start of the new tax year), you remove £20k from the ISA and put it back where you wanted it to be allocated (e.g savings or paying back margin).</p>

<p>Your ISA limit for that entire tax year (6 Apr 2019 - 5 Apr 2020) is now £40k. In fact, you should put in £40k on 5 April 2020 (e.g from savings or margin) and then take £40k back out on the next day, 6 April 2021 (the start of the new tax year) and put that back where you want it.</p>

<p>You repeat this where possible, temporarily tying up larger and larger amounts of capital (but only for a day) on the boundaries of the tax year to ensure that in each tax year, you retain the option of putting in all the previous year ISA limits.</p>

<h2 id="why-would-you-want-to-do-this">Why would you want to do this?</h2>

<p>There are a couple of reasons why you might want to do this:</p>

<ol>
  <li>You want to allocate <em>all</em> your capital to an instrument that your ISA can’t hold, but reserve the option of putting it into your ISA afterwards. As a specific example: someone who believes that paying down their mortgage faster is providing them with a better return than the stock market (given 2022/23 interest rates, this may not be as stupid as it sounds) should have an offset mortgage in which they accumulate all their capital. They should move money in and out of that offset mortgage account on the boundary between tax years to accumulate their ISA allowance and then once the mortgage is paid off, can redirect all their money to their ISA make up for past years of not filling up the ISA.</li>
  <li>You’re harvesting someone else’s allowance who you’re comfortable giving them capital for a day or two but not more than that and envision becoming comfortable doing so later down the line.</li>
</ol>

<p>The latter is the specific situation I find myself in, where I’m happy to harvest my significant others’ ISA allowance under the assumption that in the future, we’ll be combining our finances, but want to ensure that things remain seperated before then (for example before getting married v/s after getting married).</p>

<h2 id="details">Details</h2>
<ol>
  <li>Finding a provider that offers a flexible ISA is harder than it looks, they come with more complexity between the provider and HMRC (I presume) which is why they’re not the standard. As a starting point: <a href="https://www.vanguardinvestor.co.uk">Vanguard</a> have a flexible ISA.</li>
  <li>The costs for doing this should be fairly minimal. The cost for the ISA itself should be negligible if you use a percentage-based fee or fee-free provider. The cost of borrowing the capital, removing it from an offset mortage account or away from savings or any other capital allocation for a day or two should be minimal, but will grow over time.</li>
</ol>

<h4 id="disclaimer">Disclaimer</h4>

<p>The content on this site is for informational purposes only, you should not construe any such information or other material as legal, tax, investment, financial, or other advice.</p>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="finance" /><summary type="html"><![CDATA[You've probably regularly heard that your ISA allowance is 'use-it-or-lose-it' and that there is nothing more sacred than ensuring that you've used up your ISA allowance for the year because you can't carry it forward. That's true, but like all things with the UK tax system, you can game it to essentially carry forward your allowance as long as you have enough liquid capital]]></summary></entry><entry><title type="html">:moneybag: UK Income Brackets</title><link href="https://zainp.com/uk-income-brackets/" rel="alternate" type="text/html" title=":moneybag: UK Income Brackets" /><published>2021-11-25T12:00:00+00:00</published><updated>2021-11-25T12:00:00+00:00</updated><id>https://zainp.com/uk-income-brackets</id><content type="html" xml:base="https://zainp.com/uk-income-brackets/"><![CDATA[]]></content><author><name>zainpatel</name></author><category term="project" /><category term="uk" /><category term="tax" /><category term="salary" /><category term="html" /><summary type="html"><![CDATA[This website lays out the different UK income brackets, their pitfalls and benefits, what to watch out for and collates together resources for the relevant brackets on making the most of your income and eligible benefits and allowances]]></summary></entry><entry><title type="html">Run an OpenSSH server as a bastion on a Kubernetes Pod :globe_with_meridians:</title><link href="https://zainp.com/ssh-kubernetes-pod/" rel="alternate" type="text/html" title="Run an OpenSSH server as a bastion on a Kubernetes Pod :globe_with_meridians:" /><published>2021-10-19T23:00:00+00:00</published><updated>2021-10-19T23:00:00+00:00</updated><id>https://zainp.com/ssh-kubernetes-pod</id><content type="html" xml:base="https://zainp.com/ssh-kubernetes-pod/"><![CDATA[<p>I should probably caveat this up-front with the fact that this goes against the raison-d’être of Kubernetes slightly, probably isn’t best practice and you most likely shouldn’t do this in production. That said – I have this running in production and am fairly pleased with it and don’t feel <em>too</em> ashamed of myself.</p>

<h2 id="introduction">Introduction</h2>

<p>Let me outline my setup for some context. I run a Kubernetes cluster using <a href="https://aws.amazon.com/eks/">EKS</a> which runs the backend, frontend and some other small services for an application I’ve built. This cluster sits on a public subnet of my <a href="https://aws.amazon.com/vpc/">VPC</a> and my database which hosts all of the application data sits on a private subnet in <a href="https://aws.amazon.com/rds/">RDS</a>, comfortably nestled away from the world.</p>

<p>I often need access to this database, either for analytics or debugging purposes, on my local machine. This would typically call for an EC2 instance running on the same public subnet, with the appropriate security group configurations to allow it to talk to my database and let me in. Given that all of my security group configurations are done in Terraform, I don’t want to encode this ephermeral bastion into my Terraform (and I also maintain Terraform for Azure, which means more work - and I’m lazy). It just felt like the wrong level to do things at, when I had an entire EKS cluster to operate within.</p>

<p>I spent a good half-week searching (I’m a Kubernetes newbie) for how to implement the solution I had in my head, which was to run an SSH server on a Kubernetes pod (due to being within the VPC, had access to the RDS endpoint) and then use SSH local forwarding to connect to the server in that pod, and forward the RDS port down to my local machine. In an ideal world, I’d have been able to spin up an empty pod and run <code class="language-plaintext highlighter-rouge">kubectl port-forward -L 5432:endpoint:5432</code> or something. That’s a bit of a lie – the reason that that isn’t ideal is that it requires <code class="language-plaintext highlighter-rouge">kubectl</code> installed on my machine (fine for me, less so for my product manager) and also means that various tools that allow a database connection to specify an <code class="language-plaintext highlighter-rouge">ssh</code> connection no longer work (e.g PyCharm’s database viewer).</p>

<h2 id="implementation">Implementation</h2>

<p>First off, I needed to spin a pod up that runs an <a href="https://www.openssh.com">OpenSSH</a> server, preferrably with the <code class="language-plaintext highlighter-rouge">authorized_keys</code> baked into the image already. To do this I used this super simple Dockerfile from <a href="https://github.com/corbinu/ssh-server">corbinu/ssh-server</a>.</p>

<h3 id="step-1">Step 1</h3>

<p>Clone the repository down:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/corbinu/ssh-server.git
</code></pre></div></div>

<h3 id="step-2">Step 2</h3>

<p>Add the relevant public keys to the <code class="language-plaintext highlighter-rouge">authorized_keys</code> file at the root of the repository (create it if required)</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo "public_key_1" &gt; authorized_keys
echo "public_key_2" &gt;&gt; authorized_keys
</code></pre></div></div>

<p>Modify the configuration within <code class="language-plaintext highlighter-rouge">sshd_config</code> to ensure that the SSH connection doesn’t time out after 1 minute (as it does by default):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ClientAliveInterval 30
ClientAliveCountMax 5
</code></pre></div></div>

<p>for a timeout of <code class="language-plaintext highlighter-rouge">30 * 5</code> seconds. Modify as appropriate.</p>

<h3 id="step-3">Step 3</h3>

<ol>
  <li>Build the docker image, tag it with your relevant repository tag (I use Amazon <a href="https://aws.amazon.com/ecr/">ECR</a>) and push it to an appropriate registry</li>
</ol>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build -t account_id.amazon.ecr.io/repo/app-ssh-server:latest .
docker push account_id.amazon.ecr.io/repo/app-ssh-server:latest
</code></pre></div></div>

<p>Alternatively, you could leave it as-is and simply use the <code class="language-plaintext highlighter-rouge">corbinu/ssh-server</code> image from DockerHub, but will need to configure the <code class="language-plaintext highlighter-rouge">authorized_keys</code> by <code class="language-plaintext highlighter-rouge">kubectl exec -it &lt;pod-name&gt; -- /bin/bash</code> and editing the file directly, and will mean that you aren’t able to configure <code class="language-plaintext highlighter-rouge">sshd_config</code></p>

<h3 id="step-4">Step 4</h3>

<p>Spin up a Kubernetes pod with your pre-baked image:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl run app-ssh-server --image account_id.amazon.ecr.io/repo/app-ssh-server:latest --port 22 --restart Never --namespace app-maintenance
</code></pre></div></div>

<h3 id="step-5">Step 5</h3>

<p>Expose this Kubernetes pod to the outside world using a Load Balancer (your cluster must be configured to talk to some Cloud Provider to create the necessary resources):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl expose pod app-ssh-server --namespace app-maintenance --port 22
</code></pre></div></div>

<h3 id="step-6">Step 6</h3>

<p>If you monitor your services with <code class="language-plaintext highlighter-rouge">kubectl get svc -n app-maintenance</code>, you should see a service dedicated to your pod with an external IP provisoned.</p>

<p>For best security practices, edit the configuration of this created service to whitelist certain IP addresses</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl edit svc/app-ssh-server --namespace app-maintenance
</code></pre></div></div>

<p>using (for example) <code class="language-plaintext highlighter-rouge">sourceLoadBalancerRage</code> under the <code class="language-plaintext highlighter-rouge">spec</code> key.</p>

<h3 id="testing">Testing</h3>

<p>Connect to your SSH server using <code class="language-plaintext highlighter-rouge">ssh &lt;external-ip&gt;</code> and optionally forward some ports using <code class="language-plaintext highlighter-rouge">-L</code> or the <code class="language-plaintext highlighter-rouge">LocalForward</code> configuration</p>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="kubernetes" /><category term="open-source" /><category term="aws" /><summary type="html"><![CDATA[Ever needed a convenient bastion/jump-host to something hidden away in your internal private subnets? Have a Kubernetes cluster up and running already? This blog post walks you through how to spin up an OpenSSH server in a pod for easy SSH port-tunneling]]></summary></entry><entry><title type="html">:eyeglasses: Helm Template Preview</title><link href="https://zainp.com/helm-template-preview/" rel="alternate" type="text/html" title=":eyeglasses: Helm Template Preview" /><published>2021-09-08T11:00:00+00:00</published><updated>2021-09-08T11:00:00+00:00</updated><id>https://zainp.com/helm-template-preview</id><content type="html" xml:base="https://zainp.com/helm-template-preview/"><![CDATA[]]></content><author><name>zainpatel</name></author><category term="project" /><category term="open-source" /><category term="helm" /><category term="kubernetes" /><category term="go" /><summary type="html"><![CDATA[A small web-app that will accept Helm templates as well as values and render them in real-time for you, to make for easy editing of Helm templates]]></summary></entry><entry><title type="html">Increase your net income using salary sacrifice :moneybag:</title><link href="https://zainp.com/increase-your-net-with-salary-sacrifice/" rel="alternate" type="text/html" title="Increase your net income using salary sacrifice :moneybag:" /><published>2021-07-02T19:00:00+00:00</published><updated>2021-07-02T19:00:00+00:00</updated><id>https://zainp.com/increase-your-net-with-salary-sacrifice</id><content type="html" xml:base="https://zainp.com/increase-your-net-with-salary-sacrifice/"><![CDATA[<p>Too often, when people are asking what to do with their bonuses, the advice given is to divert some of it (using salary sacrifice) to their pension to make use of the tax-relief. This sounds fully correct at first glance, but on closer examination, can sometimes prove to be <em>less</em> tax efficient.</p>

<h2 id="pre-requisites">Pre-requisites</h2>

<ul>
  <li>You are employed and have your taxes operated under <a href="https://www.gov.uk/income-tax/how-you-pay-income-tax">PAYE</a> with regular monthly salary payments</li>
  <li>Your monthly gross salary is above the <a href="https://www.gov.uk/government/publications/rates-and-allowances-national-insurance-contributions/rates-and-allowances-national-insurance-contributions">Upper Earnings Limit (UEL</a> for National Insurance (NI). As of April 2021 this is currently £4189. This is roughly correlated with the higher rate tax threshold</li>
  <li>Your employer matches pension contributions holistically or anually rather than monthly</li>
  <li>Your employer operates a salary sacrifice scheme</li>
  <li>Your employer allows you to change your salary sacrifice contribution over the course of a year</li>
</ul>

<h2 id="how-does-this-work">How does this work?</h2>

<p>The fact that tax works on an annual holistic basis in the UK is well-known. It doesn’t matter when or how you earn your money, it’s taxed by PAYE in as smooth a manner as possible (hello cumulative tax codes!) but reconciled anually at the end of the tax year based on the holistic income and gains then.</p>

<p>The kicker here is that NI is calculated per pay-period and not reconciled anually. There’s not quite the same thing as “over paying” NI in one month due to a bonus and expecting a rebate at the end of the year or in next months payslip as there is with tax.</p>

<p>Additionally, salary sacrifice works by reducing your salary in exchange for your employer contributing directly to your pension. In HMRC’s eyes, it’s as if you never earnt that money. The timing of this doesn’t make any difference for PAYE because the only figure that matters is your total taxable income at the end of the tax year. It does make a difference for NI, however.</p>

<h3 id="the-typical-situation">The typical situation</h3>

<p>A typical agreement between an employee and employer will have the employee agree to <code class="language-plaintext highlighter-rouge">x%</code> salary sacrifice as a pension contribution which is applied to each monthly payslip. As an example, take Bob who earns £96k and salary sacrifices 20% to his pension. His monthly payslip looks like:</p>

<ul>
  <li>Gross: £8,000</li>
  <li>Pension contribution (SS): £1,600</li>
  <li>Taxable income: £6,400</li>
  <li>Tax: £1,512.66</li>
  <li>NI: £451.26 (<code class="language-plaintext highlighter-rouge">(6400 - 4189) * 2/100 + (4189 - 797) * 12/100</code>)</li>
  <li>Net: £4426.08</li>
</ul>

<p>Had he not made any pension contributions, it would look like:</p>

<ul>
  <li>Gross: £8,000</li>
  <li>Pension contribution (SS): £0</li>
  <li>Taxable income: £8,000</li>
  <li>Tax: £2152.66</li>
  <li>NI: £483.26 (<code class="language-plaintext highlighter-rouge">(8000 - 4189) * 2/100 + (4189 - 797) * 12/100</code>)</li>
  <li>Net: £5364.08</li>
</ul>

<p>you can see that the salary sacrifice has reduced the tax paid substantially (all at 40%) whereas has barely impacted the amount of NI paid (most of it as 2%). This is because the NI payment drops from 12% to 2% when you cross the upper earning threshold, so you’re getting <em>less</em> relief.</p>

<h3 id="shuffling-things-around">Shuffling things around</h3>

<p>Given that tax is paid holistically, we don’t care about the monthly tax position, it’ll sort itself out at the end of the year, but we can concentrate our pension contributions to a single (or as few as possible) month(s) to make sure that we make full use of the 12% NI relief rather than the 2%.</p>

<p>Bob makes a total pension contribution of £19,200. If we split that across three months instead of 12 (we need each monthly payslip to be above the minimum wage for a months full-time work), those three months look like:</p>

<ul>
  <li>Gross: £8000</li>
  <li>Pension contribution (SS): £6,400</li>
  <li>Taxable income: £1,600</li>
  <li>Tax: (doesn’t matter)</li>
  <li>NI: £96.36</li>
</ul>

<p>and the rest of the 9 months look like the situation above where he doesn’t make any pension contributions whatsoever.</p>

<h3 id="the-result">The result</h3>

<p>In both situations, Bob pays the exact same amount of tax (£18,152) by the end of the year but pays £5415.12 (<code class="language-plaintext highlighter-rouge">451.26 * 12</code>) NI by paying it monthly and £4,638.42 (<code class="language-plaintext highlighter-rouge">483.26 * 9 + 96.36 * 3</code>) by concentrating the contributions to 3 months.</p>

<p>That means Bob puts the exact same amount of money into his pension at the end of the year, pays the exact same amount of tax, but sees an extra £776.70 show up in his bank account for doing nothing other than pestering his employers HR to follow his ‘convoluted’ pension contribution strategy. This saving comes from exploiting the fact that NI is calculated per pay-period and maximising the relief at 12% from a few months rather than skimming the 2% relief every month.</p>

<h2 id="caveats">Caveats</h2>

<p>This ignores:</p>

<ul>
  <li>employer contributions which you might be missing out on depending on how your employer calculates their matching (mine is non-contributory, so I don’t miss out on anything there)</li>
  <li>A cash-flow crunch, there are 3 months in which you see much less come in from your payslip which you need to conciously cover from the increased monthly payments in other months. You could also price in the time-value of money (though that is <em>very</em> nitpicky)</li>
  <li>If you choose to back-load the pension then you miss out on ‘time in the market’ and potential gains, but ensure that you remove the risk of ‘over-contributing to your pension’ early-on and removing flexibility had you been let-go or some emergency come up since pension contributions are irrecoverable</li>
  <li>If you choose to front-load the pension, then you gain ‘time in the market’ which is a good thing but lose out on flexibility (say an emergency comes up later in the year where you wish you could then turn your contribution down).</li>
</ul>

<p>The last 3 points can be mitigated somewhat by spreading Bob’s three months across the tax year, but may not be possible depending on your level of pension contribution and salary (maybe only one month is enough to make your full pension contribution).</p>

<h2 id="conclusion">Conclusion</h2>

<p>If the stars align just right and this works for your situation as it does mine, then this can be a nice chunk of free money for almost no work and I would definitely recommend doing it, though do have a chat with your HR department before to make sure that it’s all kosher for you.</p>

<h4 id="disclaimer">Disclaimer</h4>

<p>The content on this site is for informational purposes only, you should not construe any such information or other material as legal, tax, investment, financial, or other advice.</p>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="finance" /><summary type="html"><![CDATA[Increasing your net income using a salary sacrifice (deducting from your salary) sounds unintuitive (for good reason!) but there are a small niche set of scenarios where it does work. This exploits the fact that National Insurance is calculated per pay-period rather than holistically over the tax year as tax is]]></summary></entry><entry><title type="html">:mailbox: Kedro Local Notify</title><link href="https://zainp.com/kedro-local-notify/" rel="alternate" type="text/html" title=":mailbox: Kedro Local Notify" /><published>2021-02-10T01:00:00+00:00</published><updated>2021-02-10T01:00:00+00:00</updated><id>https://zainp.com/kedro-local-notify</id><content type="html" xml:base="https://zainp.com/kedro-local-notify/"><![CDATA[]]></content><author><name>zainpatel</name></author><category term="project" /><category term="python" /><category term="kedro" /><category term="macos" /><summary type="html"><![CDATA[Lightweight plugin that will trigger a native local notification to popup when a Kedro pipeline run finishes sucesfully or fails, with some useful information in it.]]></summary></entry><entry><title type="html">:fishing_pole_and_fish: Kedro Dataframe Drop-in</title><link href="https://zainp.com/kedro-dataframe-dropin/" rel="alternate" type="text/html" title=":fishing_pole_and_fish: Kedro Dataframe Drop-in" /><published>2021-01-14T23:00:00+00:00</published><updated>2021-01-14T23:00:00+00:00</updated><id>https://zainp.com/kedro-dataframe-dropin</id><content type="html" xml:base="https://zainp.com/kedro-dataframe-dropin/"><![CDATA[]]></content><author><name>zainpatel</name></author><category term="project" /><category term="python" /><category term="kedro" /><category term="gpu" /><category term="rapids" /><category term="dask" /><summary type="html"><![CDATA[Provides a variety of alternative datasets to Kedro's suite of pandas datasets, with support for `modin[ray]`, `modin[dask]`, `cudf`, `dask` and `dask-cudf` with the intention of allowing Kedro users to accelerate their workflows using GPUs or clusters of GPUs.]]></summary></entry><entry><title type="html">Add ordering to your Django-Graphene GraphQL API :arrow_down:</title><link href="https://zainp.com/Add-ordering-django-graphql/" rel="alternate" type="text/html" title="Add ordering to your Django-Graphene GraphQL API :arrow_down:" /><published>2020-10-31T20:00:00+00:00</published><updated>2020-10-31T20:00:00+00:00</updated><id>https://zainp.com/Add-ordering-django-graphql</id><content type="html" xml:base="https://zainp.com/Add-ordering-django-graphql/"><![CDATA[<h2 id="the-problem">The problem</h2>

<p>The canonical <a href="https://docs.graphene-python.org/projects/django/en/latest/filtering/#ordering">Djanago-Graphene documentation on ordering</a> points you towards using the <code class="language-plaintext highlighter-rouge">OrderingFilter</code> on a custom <code class="language-plaintext highlighter-rouge">FilterSet</code> from <code class="language-plaintext highlighter-rouge">django_filter</code> to implement ordering on your API, so that you can do things like:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>query {
    posts(orderBy: "-createdAt") {
        title
    }
}
</code></pre></div></div>

<p>to get post titles ordered in descending order of when they were created. This works okay, as long as you don’t intend to use the filtering mechanism in <code class="language-plaintext highlighter-rouge">django-graphene</code>/<code class="language-plaintext highlighter-rouge">django-filter</code>, where you can specify django-style filters that are converted automatically to GraphQL arguments for filtering:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">PostNode</span><span class="p">(</span><span class="n">DjangoObjectType</span><span class="p">):</span>
    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
        <span class="n">fields</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">"title"</span><span class="p">:</span> <span class="p">[</span><span class="s">"exact"</span><span class="p">,</span> <span class="s">"startswith"</span><span class="p">,</span> <span class="s">"icontains"</span><span class="p">]</span>
        <span class="p">}</span>
</code></pre></div></div>

<p>because as soon as you follow the documentation from <code class="language-plaintext highlighter-rouge">django-graphene</code> and provide the custom filterset class, you gain ordering via the <code class="language-plaintext highlighter-rouge">orderBy</code>, but lose the default <code class="language-plaintext highlighter-rouge">FilterSet</code> that <code class="language-plaintext highlighter-rouge">django-graphene</code> builds for your <code class="language-plaintext highlighter-rouge">PostNode</code> type, which contains the nice <code class="language-plaintext highlighter-rouge">title__Exact</code>, <code class="language-plaintext highlighter-rouge">title__Icontains</code>, etc… filters on it.</p>

<h2 id="the-solution">The solution</h2>

<p>To fix this, I built a custom connection field, inspired from <a href="https://stackoverflow.com/questions/57478464/django-graphene-relay-order-by-orderingfilter">several stackoverflow answers</a> inheriting from <code class="language-plaintext highlighter-rouge">DjangoFilterConnectionField</code> that you likely already use:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">graphene_django.filter</span> <span class="kn">import</span> <span class="n">DjangoFilterConnectionField</span>
<span class="kn">from</span> <span class="nn">graphene.utils.str_converters</span> <span class="kn">import</span> <span class="n">to_snake_case</span>


<span class="k">class</span> <span class="nc">OrderedDjangoFilterConnectionField</span><span class="p">(</span><span class="n">DjangoFilterConnectionField</span><span class="p">):</span>
    <span class="o">@</span><span class="nb">classmethod</span>
    <span class="k">def</span> <span class="nf">resolve_queryset</span><span class="p">(</span>
        <span class="n">cls</span><span class="p">,</span> <span class="n">connection</span><span class="p">,</span> <span class="n">iterable</span><span class="p">,</span> <span class="n">info</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">filtering_args</span><span class="p">,</span> <span class="n">filterset_class</span>
    <span class="p">):</span>
        <span class="n">qs</span> <span class="o">=</span> <span class="nb">super</span><span class="p">().</span><span class="n">resolve_queryset</span><span class="p">(</span>
            <span class="n">connection</span><span class="p">,</span> <span class="n">iterable</span><span class="p">,</span> <span class="n">info</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">filtering_args</span><span class="p">,</span> <span class="n">filterset_class</span>
        <span class="p">)</span>
        <span class="n">order</span> <span class="o">=</span> <span class="n">args</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"orderBy"</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">order</span><span class="p">:</span>
            <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">order</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
                <span class="n">snake_order</span> <span class="o">=</span> <span class="n">to_snake_case</span><span class="p">(</span><span class="n">order</span><span class="p">)</span>
            <span class="k">else</span><span class="p">:</span>
                <span class="n">snake_order</span> <span class="o">=</span> <span class="p">[</span><span class="n">to_snake_case</span><span class="p">(</span><span class="n">o</span><span class="p">)</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">order</span><span class="p">]</span>

            <span class="c1"># annotate counts for ordering
</span>            <span class="k">for</span> <span class="n">order_arg</span> <span class="ow">in</span> <span class="n">snake_order</span><span class="p">:</span>
                <span class="n">order_arg</span> <span class="o">=</span> <span class="n">order_arg</span><span class="p">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s">"-"</span><span class="p">)</span>
                <span class="n">annotation_name</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"annotate_</span><span class="si">{</span><span class="n">order_arg</span><span class="si">}</span><span class="s">"</span>
                <span class="n">annotation_method</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">qs</span><span class="p">,</span> <span class="n">annotation_name</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
                <span class="k">if</span> <span class="n">annotation_method</span><span class="p">:</span>
                    <span class="n">qs</span> <span class="o">=</span> <span class="n">annotation_method</span><span class="p">()</span>

            <span class="c1"># override the default distinct parameters
</span>            <span class="c1"># as they might differ from the order_by params
</span>            <span class="n">qs</span> <span class="o">=</span> <span class="n">qs</span><span class="p">.</span><span class="n">order_by</span><span class="p">(</span><span class="o">*</span><span class="n">snake_order</span><span class="p">).</span><span class="n">distinct</span><span class="p">()</span>

        <span class="k">return</span> <span class="n">qs</span>
</code></pre></div></div>

<p>and you use it like so in your query schema:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">PostQuery</span><span class="p">(</span><span class="n">graphene</span><span class="p">.</span><span class="n">ObjectType</span><span class="p">):</span>
    <span class="n">posts</span> <span class="o">=</span> <span class="n">OrderedDjangoFilterConnectionField</span><span class="p">(</span>
        <span class="n">PostNode</span><span class="p">,</span>
        <span class="n">orderBy</span><span class="o">=</span><span class="n">graphene</span><span class="o">=</span><span class="n">graphene</span><span class="p">.</span><span class="n">List</span><span class="p">(</span><span class="n">of_type</span><span class="o">=</span><span class="n">graphene</span><span class="p">.</span><span class="n">String</span><span class="p">)</span>
    <span class="p">)</span>
</code></pre></div></div>

<p>This differs from the StackOverflow answer in that it lets you sort by custom fields that you may have implemented on your GraphQL API, but not on your model (in my case, it was the dowmload count of a post - which isn’t stored on the model) as long as you build a relevant annotation method for it. Let’s walk through what this does:</p>

<ul>
  <li>We call the parent <code class="language-plaintext highlighter-rouge">DjangoFilterConnectionField</code>’s <code class="language-plaintext highlighter-rouge">resolve_queryset</code> method to get the queryset with all the magic filtering taken care of already, this ensures we don’t have to do any work ourselves or re-build any logic.</li>
  <li>If the GraphQL query is sent <em>without</em> the <code class="language-plaintext highlighter-rouge">orderBy</code> argument then we never enter the  <code class="language-plaintext highlighter-rouge">if order</code> block and instantly return the <code class="language-plaintext highlighter-rouge">queryset</code> as-is from above.</li>
  <li>If the GraphQL query is sent <em>with</em> the <code class="language-plaintext highlighter-rouge">orderBy</code> argument (say <code class="language-plaintext highlighter-rouge">fieldName</code>) then we convert this to snake case (<code class="language-plaintext highlighter-rouge">field_name</code>) and do the following (note: if the value is instead a list, we convert each value to snake case and do the following to each value)</li>
  <li>We look at the queryset for a method called <code class="language-plaintext highlighter-rouge">annotate_{field_name}</code> on it
    <ul>
      <li>If this method exists, we call it, expecting the result to be the same queryset-type, but with an additional annotation on it called <code class="language-plaintext highlighter-rouge">field_name</code> (this is what lets us do the ordering) and replace the queryset with the newly-annotated queryset.</li>
      <li>If the method doesn’t exist, then we leave the queryset as-is and do continue with the rest of the logic</li>
    </ul>
  </li>
  <li>We call the <code class="language-plaintext highlighter-rouge">order_by</code> argument on the queryset we have with the values from the <code class="language-plaintext highlighter-rouge">orderBy</code> snake-cased arguments provided, this works on native db fields as normal with Django and also works with any graphene fields you’ve built that have custom resolvers as long as you write a mathching annotation method on the queryset for that field.</li>
  <li>We also call <code class="language-plaintext highlighter-rouge">distinct</code> on the queryset to ensure we get sensical results, given some of Django’s quirks with <code class="language-plaintext highlighter-rouge">order_by</code> and <code class="language-plaintext highlighter-rouge">ManyToMany</code> fields.</li>
</ul>

<p>This works as you would expect if you were ordering by a model field, say the post title, but the real beauty is in ordering on a custom field you’ve created on your GraphQL API, but not on the model, in my case the post download count. This is what the <code class="language-plaintext highlighter-rouge">annotate_field_name</code> method is for. My Post Manager looks like:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">PostManager</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Manager</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">annotate_download_count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">annotate</span><span class="p">(</span><span class="n">download_count</span><span class="o">=</span><span class="n">models</span><span class="p">.</span><span class="n">Count</span><span class="p">(</span><span class="s">"downloads"</span><span class="p">))</span>
</code></pre></div></div>

<p>which annotates my queryset with a <code class="language-plaintext highlighter-rouge">download_count</code> field, that I can then order against.</p>

<p>You’d also ideally re-use this annotation within your GraphQL resolver rather than writing any new logic. For example:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">PostNode</span><span class="p">:</span>
    <span class="n">download_count</span> <span class="o">=</span> <span class="n">graphene</span><span class="p">.</span><span class="n">Int</span><span class="p">()</span>

    <span class="o">@</span><span class="nb">staticmethod</span>
    <span class="k">def</span> <span class="nf">resolve_download_count</span><span class="p">(</span><span class="n">root</span><span class="p">:</span> <span class="n">Post</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
        <span class="k">return</span> <span class="p">(</span>
            <span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="nb">filter</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">root</span><span class="p">.</span><span class="nb">id</span><span class="p">)</span>
            <span class="p">.</span><span class="n">annotate_download_count</span><span class="p">()</span>
            <span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">root</span><span class="p">.</span><span class="nb">id</span><span class="p">)</span>
            <span class="p">.</span><span class="n">download_count</span>
        <span class="p">)</span>

</code></pre></div></div>

<p>or if you want to optimise things when resolving multiple posts (e.g to list them all):</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># schema.py
</span>
<span class="kn">from</span> <span class="nn">.types</span> <span class="kn">import</span> <span class="n">PostNode</span>

<span class="k">class</span> <span class="nc">PostQuery</span><span class="p">(</span><span class="n">graphene</span><span class="p">.</span><span class="n">ObjectType</span><span class="p">):</span>
    <span class="n">posts</span> <span class="o">=</span> <span class="n">OrderedDjangoFilterConnectionField</span><span class="p">(</span><span class="n">PostNode</span><span class="p">)</span>

    <span class="o">@</span><span class="nb">staticmethod</span>
    <span class="k">def</span> <span class="nf">resolve_posts</span><span class="p">():</span>
        <span class="k">return</span> <span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="nb">all</span><span class="p">().</span><span class="n">annotate_download_count</span><span class="p">()</span>

<span class="c1"># types.py
</span>
<span class="k">class</span> <span class="nc">PostNode</span><span class="p">:</span>
    <span class="n">download_count</span> <span class="o">=</span> <span class="n">graphene</span><span class="p">.</span><span class="n">Int</span><span class="p">()</span>

    <span class="o">@</span><span class="nb">staticmethod</span>
    <span class="k">def</span> <span class="nf">resolve_download_count</span><span class="p">(</span><span class="n">root</span><span class="p">:</span> <span class="n">Post</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">root</span><span class="p">.</span><span class="n">download_count</span> <span class="c1"># has been annotated from the `posts` resolver
</span></code></pre></div></div>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="python" /><category term="django" /><category term="graphql" /><summary type="html"><![CDATA[Django-Graphene provides an OrderingFilter that doesn't really work if you use any other filtering features with Django. This post walks through how I worked around this and implemented an extensible ordering mechanism on my Django GraphQL API, including the ability to filter on custom fields (as opposed to only model fields)]]></summary></entry><entry><title type="html">How to play Among Us on your Macbook :video_game:</title><link href="https://zainp.com/Play-among-us-on-your-mac/" rel="alternate" type="text/html" title="How to play Among Us on your Macbook :video_game:" /><published>2020-10-02T18:00:00+00:00</published><updated>2020-10-02T18:00:00+00:00</updated><id>https://zainp.com/Play-among-us-on-your-mac</id><content type="html" xml:base="https://zainp.com/Play-among-us-on-your-mac/"><![CDATA[<p><img src="https://steamcdn-a.akamaihd.net/steam/apps/945360/header.jpg" alt="" /></p>

<p>If you’re anything like me, then you’ve got yourself a Macbook (or 2 - definitely not using my work laptop to play games, no, not me! :eyes:) and are loathe to dual boot Windows via Bootcamp or spin up a heavy VM via Parallels just to be able to play Among Us. This posts is a quick guide to get Among Us running on your laptop/computer running OSx. I’m running on the latest Catalina version and haven’t tested this with any previous versions of OSx.</p>

<h2 id="1-buy-among-us-on-steam">1. Buy Among Us on Steam</h2>

<p>The creators of the game deserve that sweet Steam revenue for the awesome game they’ve created. This is different to some of the other approaches out there on the web that get you to play Among Us on your Mac via Bluestacks (which frequently crashes) and uses the free Among Us mobile client via the Android Emulator.</p>

<p>Also… <em>skins</em>.</p>

<p><em>Note</em>: You won’t be able to install Among Us on Steam, since you’re running on OSx and it’s only been built for Windows. This is okay and to be expected.</p>

<p><em>Note 2</em>: Buying it on Steam also means that we’re likely to get the Mac client if and when the Among Us creators decide to update Among Us to have one (unlikely with all the work they have cut out for them though)</p>

<h2 id="2-download-playonmac">2. Download PlayOnMac</h2>

<p>Install <a href="https://www.playonmac.com/en/">PlayOnMac</a> - it’s roughly 600mb to download and installs/unzips to about 2gb, which is a little larger than what I was expecting.</p>

<p>From what I can tell, PlayOnMac bundles together some kind of wine emulator/client that runs on OSx - get ready for lots of windows-style dialogue pop ups, bringing back some nostalgic and decidedly creepy feels seeing it on your Macbook screen.</p>

<p>When opening PlayOnMac for the first time, you may get a popup (if you’re on Catalina) that will deny you access to open the application. You can get past this by going to Security in your System Preferences and clicking “Open anyway” on the “General” tab.</p>

<h2 id="3-install-windows-steam-on-playonmac">3. Install Windows-Steam on PlayOnMac</h2>

<p>Launch the PlayOnMac application and press “Install” on the GUI that opens up. Open the “Games” tab on the top and search for <em>Steam</em> (not Among Us!). Hit install and run through the installer.</p>

<p>If you get anything about Wine asking you to install Gecko for “embedded HTML”, feel free to cancel - it isn’t relevant to us.</p>

<p>This should install Steam in <code class="language-plaintext highlighter-rouge">/Users/&lt;user&gt;/Library/PlayOnMac/wine_prefix/Steam</code> which is a virtual drive that mirrows Windows structure of <code class="language-plaintext highlighter-rouge">Program Files (x86)/Steam/steam.exe</code>.</p>

<h2 id="4-open-up-steam">4. Open up Steam</h2>

<p>You’ll need to login to Steam, likely pasting in a security code from your email and Steam should then open up. You might see a blank screen on the Steam GUI. If you do, close Steam and exit PlayOnMac and follow the next step.</p>

<h2 id="5-fix-steam">5. Fix Steam</h2>

<p>Open up PlayOnMac again, left-click on Steam and then click “Configure” in the sidebar that pops up on the left of the GUI. In the “General” tab of the configuration editor, paste the following in the “Arguments” field: <code class="language-plaintext highlighter-rouge">wine steam.exe -no-browser +open steam://open/minigameslist</code> which ensures that Steam only opens the mini-Library list that is compatible with Wine.</p>

<h2 id="6-re-open-steam">6. Re-open Steam</h2>

<p>Re-open Steam (by hitting “Run”) and go ahead and install Among Us, it’s a 50mb download and 200mb unzipped and installed, so should be relatively quick. You can now run the game!</p>

<p>Don’t forget to switch the controls from “Mouse” to “Mouse+Keyboard”.</p>

<h2 id="7-red-sus">7. Red sus</h2>

<p><img src="https://vignette.wikia.nocookie.net/among-us-wiki/images/a/a6/1_red.png/" alt="" /></p>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="osx" /><category term="game" /><summary type="html"><![CDATA[Among Us has been recently re-kindled and is taking the world by storm, including my group of friends. This post is a quick guide to get Among Us working on your Mac.]]></summary></entry><entry><title type="html">Github’s secret new user README’s :octocat:</title><link href="https://zainp.com/Githubs-secret-new-README/" rel="alternate" type="text/html" title="Github’s secret new user README’s :octocat:" /><published>2020-07-09T22:00:00+00:00</published><updated>2020-07-09T22:00:00+00:00</updated><id>https://zainp.com/Githubs-secret-new-README</id><content type="html" xml:base="https://zainp.com/Githubs-secret-new-README/"><![CDATA[<p>From what I gather, it looks like Github covertly launched a new feature that allows users to create a <code class="language-plaintext highlighter-rouge">README.md</code> that renders on their user profile page, which is a pretty big shakeup of the github profile page, in my opinion. I heard about it through one of my friends, who heard about it from a friend, who heard about it on Twitter.</p>

<p>This is what it looks like (from <a href="https://github.com/mzjp2">my current profile</a>):</p>

<p><img src="https://imgur.com/I06ZD1l.png" alt="mzjp2 github profile screenshot" /></p>

<p>To get your own, you need to create a new repository in your account with the same name as your account, so in my case, with my account name being <a href="https://github.com/mzjp2">mzjp2</a> I needed to create a <em>public</em> repository called <code class="language-plaintext highlighter-rouge">mzjp2</code>, accessible at <a href="https://github.com/mzjp2/mzjp2">github.com/mzjp2/mzjp2</a> with, at least, a <code class="language-plaintext highlighter-rouge">README.md</code>.</p>

<p><img src="https://imgur.com/meBGUIz.png" alt="new repo creation screen" /></p>

<p>You can add additional files to your repository (I have a folder of <code class="language-plaintext highlighter-rouge">svg</code> icons located in there that I use as images in my <code class="language-plaintext highlighter-rouge">README</code>), but the only thing that ultimately gets rendered is the <code class="language-plaintext highlighter-rouge">README.md</code>. It follows the same rendering rules and follows the same Github markdown spec as all the other <code class="language-plaintext highlighter-rouge">README</code>’s you’re used to in any other repo.</p>

<p>One caveat that I ran into is that you shouldn’t use relative links in the <code class="language-plaintext highlighter-rouge">README</code>. For example, I had an <code class="language-plaintext highlighter-rouge">&lt;img src="icons/linkedin.svg"&gt;</code> reference in my <code class="language-plaintext highlighter-rouge">README.md</code> which rendered fine locally since my repository structure had <code class="language-plaintext highlighter-rouge">icons/</code> and <code class="language-plaintext highlighter-rouge">README.md</code> on the same level <em>and</em> rendered fine on the <code class="language-plaintext highlighter-rouge">README</code> view inside the <code class="language-plaintext highlighter-rouge">mzjp2/mzjp2</code> repo, but did <em>not</em> render well on my Github user profile. I switched to using aboslute links (pointing at Github’s user content static URL) and everything worked fine after that.</p>

<p>As you can imagine, people have gone wild with this new feature, building everything from a guestbook to a communal chess game on their profile. Some of the ones I’ve seen so far:</p>

<ul>
  <li><a href="https://github.com/WaylonWalker">github.com/WaylonWalker</a> that uses Netlify’s <code class="language-plaintext highlighter-rouge">_redirect</code> feature to always point to his latest blog post on his README and also incorporated the clever <code class="language-plaintext highlighter-rouge">visitor</code> badge that I shamelessly stole.</li>
  <li><a href="https://github.com/timburgan">github.com/timburgan</a> a Github staff member (I wonder how much time he had to play around with this :thinking:) that built a community chess tournament using buttons (to move your chess piece) that opens a Github issue prepopulated text, which triggers a Github action that updates the <code class="language-plaintext highlighter-rouge">README.md</code> file with the new state of the game. :clap:</li>
  <li><a href="https://github.com/saadeghi">github.com/saadeghi</a> who has the infamous Chrome dino game as a GIF playing. Minimalism, anyone?</li>
</ul>

<p>and plenty more. If there are any interesting ones you’ve come across, feel free to ping me an email and I’d love to check it out!</p>]]></content><author><name>zainpatel</name></author><category term="blog" /><category term="github" /><category term="feature" /><summary type="html"><![CDATA[Github covertly launched README's for user profiles. Here's how to get your own and inspire yourself from others.]]></summary></entry></feed>