Upload folder using huggingface_hub

2003.00001v1.mmd

@@ -0,0 +1,260 @@
+# The mathematics of Bitcoin
+
+Cyril Grunspan (De Vinci Research Center, Paris, France)
+
+Ricardo Perez-Marco (CNRS, IMJ-PRG, Sorbonne Universite, Paris, France)
+
+###### Abstract
+
+Bitcoin is a new decentralized payment network that started operating in January 2009. This new technology was created by a pseudonymous author, or group of authors, called Satoshi Nakamoto in an article that was publically released [1] in the cypherpunk mailing list. The cypherpunks are anarchists and cryptographers that who have been concerned with personal privacy in the Internet since the 90's. This article follows on a general presentation of Bitcoin by the second author [2]. We refer to this previous article for general background. Here we focus on mathematics being a feature of the security and effectiveness of Bitcoin protocol.
+
+Roughly speaking the Bitcoin's protocol is a mathematical algorithm on a network which manages transaction data and builds majority consensus among the participants. Thus, if a majority of the participants are honest, then we get an honest _automatic_ consensus. Its main feature is _decentralization_, which means that no organization or central structure is in charge. The nodes of the network are voluntary participants that enjoy equal rights and obligations. The network is open and anyone can participate. Once launched the network is resilient and unstopable. It has been functioning permanently without significant interruption since january 2009.
+
+The code and development are open. The same code has been reused and modified to create hundreds of other cryptocurrencies based on the same principles. The security of the network relies on strong cryptography (several orders of magnitude stronger than the cryptography used in classical financial services). For example, classical hash functions (SHA256, RIPEMD-160) and elliptic curve digital signatures algorithm (EDSA) are employed. The cryptography used is very standard and well known, so we will dwell on the mathematics of these cryptographic tools, but interesting cryptographical research is motivated by the special features of other cryptocurrencies.
+
+The Bitcoin network is composed by nodes, that correspond to the Bitcoin program running on different machines, that communicate with their neighbours. Properly formatted Bitcoin transactions flood the network, and are checked, broadcasted and validated continuously by the nodes which follow a precise set of rules. There is no way to force the nodes to follow these rules. Incentives are created so that any divergence from the rules is economically penalised, thus creating a virtuous cycle. In this way, the network is a complex Dynamical System and it is far from obvious that it is stable. The stability of this system is a very interesting and fundamental mathematical problem. In its study we will encounter special functions, martingale theory, Markov chains, Dyck words, etc.
+
+Nodes in the network broadcast transactions and can participate in their validation. The process of validating transactions is also called "mining" because it is related to the production of new bitcoins. The intuition behind Bitcoin is that of a sort of "electronic gold" and the rate of production of bitcoins is implemented in the protocol rules. On average every 10 minutes a block of transactions is validated and new bitcoins are minted in a special transaction without bitcoin input, called the coinbase transaction. At the beginning 50 \(\lx@math@degree\) were created in each block, and about every 4 years (more precisely, every 210 000 blocks), the production is divided by 2. This event is called a "halving". So far, we had two halvings, and the production is currently set at 12.5 \(\lx@math@degree\) per 10 minutes, or 1 800 \(\lx@math@degree\) per day. The next halving will occur on May 2020. This geometric decrease of the production limits the total amount of bitcoins to 21 millions. Currently, about 18 millions have already been created. Each block containing the validated transactions can contain about 3 to 4 thousand transactions and has a size of about 1 Mb. These blocks are linked together cryptographically, and the set of all these blocks forms the "blockchain" that contains the full history of Bitcoin transactions. This data is stored efficiently, and the current blockchain is only about 260.000 Mb. The cryptographical link between blocks is provided by the mining/validation procedure that is based on a hash function and a "Proof-of-Work". It costs computation power to validate a block and this is what ensures that the data cannot be tampered or corrupted. In order to modify a single bit of a block, we must redo all computations that has been used to build all the subsequent blocks until the last current one. Currently the computation power needed to change only the last few blocks of the more than 600 thousands composing the blockchain is beyond the capabilities of any state or company.
+
+The mining/validation procedure is a sort of decentralized lottery. A miner (this is a node engaging in validating transactions) packs together a block of floating not yet validated transactions, and builds a header of this block that contains a hash of the previous block header. The hash algorithm used is SHA-256 (iterated twice) that outputs 256 bits. Mathematically, a hash function is a determinis
+
+Figure 1: The Bitcoin logo.
+
+It is easy to compute, but practically impossible to find preimages, or collisions (two files giving the same output). Also it enjoys pseudo-random properties, that is, if we change a bit of the input, the bits of the output behave as uncorrelated random variables taking the values 0 and 1 with equal probabilities. The mining procedure consists of finding a hash that is below a pre-fixed threshold, which is called the _difficulty_. The difficulty is updated every two weeks (or more precisely every 2016 blocks) so that the rate of validation remains at 1 block per 10 minutes. The pseudo-random properties of the hash function ensure that the only way to find this hash is to probe many hashes by changing a parameter in the header (the nonce). The first miner to find a solution makes the block public, and the network adopts the block as the last block in the blockchain.
+
+It can happen that two blocks are simultaneously validated in different parts of the network. Then a competition follows between the two candidates, and the first one to have a mined block on top of it wins. The other one is discarded and is called an _orphan_ block. The blockchain with the larger amount of work (which is in general the longer one) is adopted by the nodes.
+
+When a transaction is included in the last block of the blockchain we say that it has one confirmation. Any extra block mined on top of this one gives another confirmation to the transaction and engraves it further inside the blockchain.
+
+This apparently complicated procedure is necessary to ensure that neither the network nor the blockchain cannot be corrupted. Any participant must commit some computer power in order to participate in the decision of validation. The main obstacle for the invention of a decentralised currency was to prevent double spend without a central accounting authority. Hence, the first mathematical problem than Nakamoto considers in his founding article [1] is to estimate the probability of a double spend. In the following we consider this and other stability problems, and prove mathematically the (almost general) stability of the mining rules.
+
+## 2 The mining model.
+
+We consider a miner with a fraction \(0<p\leq 1\) of the total hashrate. His profit comes from the block rewards of his validated blocks. It is important to know the probability of success when mining a block. The average number of blocks per unit of time that he succeeds mining is proportional to his hashrate \(p\). The whole network takes on average \(\tau_{0}=10\) min to validate a block, hence our miner takes on average \(t_{0}=\frac{\tau_{0}}{p}\). We consider the random variable \(\mathbf{T}\) giving the time between blocks mined by our miner. The pseudo-random properties of the hash function shows that mining is a Markov process, that is, memoryless. It is then an elementary exercise to show from this property that \(\mathbf{T}\) follows an exponential distribution,
+
+\[f_{\mathbf{r}}(t)=\alpha e^{-\alpha t}\]
+
+where \(\alpha=1/t_{0}=1/\mathbb{E}[\mathbf{T}]\). If the miner starts mining at \(t=0\), and if we denote \(\mathbf{T}_{1}\) the time needed to mine a first block, then \(\mathbf{T}_{2},\ldots,\mathbf{T}_{n}\) the inter-block mining times of successive blocks, then the Markov property shows that the random variables \(\mathbf{T}_{1},\mathbf{T}_{2},\ldots,\mathbf{T}_{n}\) are independent and are all identically distributed following the same exponential law. Therefore, the time needed to discover \(n\) blocks is
+
+\[\mathbf{S}_{n}=\mathbf{T}_{1}+\mathbf{T}_{2}+\ldots+\mathbf{T}_{n}\;.\]
+
+The random variable \(\mathbf{S}_{n}\) follows the \(n\)-convolution of the exponential distribution and, as is well known, this gives a Gamma distribution with parameters \((n,\alpha)\),
+
+\[f_{\mathbf{S}_{n}}(t)=\frac{\alpha^{n}}{(n-1)!}t^{n-1}e^{-\alpha t}\]
+
+with cumulative distribution
+
+\[F_{\mathbf{S}_{n}}(t)=\int_{0}^{t}f_{\mathbf{S}_{n}}(u)du=1-e^{-\alpha t} \sum_{k=0}^{n-1}\frac{(\alpha t)^{k}}{k!}\;.\]
+
+From this we conclude that if \(\mathbf{N}(t)\) is the process counting the number of blocks validated at time \(t>0\), \(\mathbf{N}(t)=\max\{n\geq 0;\mathbf{S}_{n}<t\}\), then we have
+
+\[\mathbb{P}[\mathbf{N}(t)=n]=F_{\mathbf{S}_{n}}(t)-F_{\mathbf{S}_{n+1}}(t)= \frac{(\alpha t)^{n}}{n!}\;e^{-\alpha t}\;,\]
+
+and \(\mathbf{N}(t)\) follows a Poisson law with mean value \(\alpha t\). This result is classical, and the mathematics of Bitcoin mining, as well as other crypto-currencies with validation based on proof of work, are Poisson distributions.
+
+## 3 The double spend problem.
+
+The first crucial mathematical problem that deserves attention in the Bitcoin protocol is the possibility of realisation of a double spend. This was the major obstacle to overcome for the invention of decentralized cryptocurrencies, thus it is not surprising that Nakamoto addresses this problem in Section 11 of his founding article [1]. He considers the situation where a malicious miner makes a payment, then in secret tries to validate a second conflicting transaction in a new block, from the same address, but to a new address that he controls, which allows him to recover the funds.
+
+For this, once the first transaction has been validated in a block in the official blockchain and the vendor delivered the goods (the vendor will not deliver unless some confirmations are visible), the only possibility consists in rewriting the blockchain from that block. This is feasible if he controls a majority of the hashrate, that is, if his relative hashrate \(q\) satisfies \(q>1/2\), because then he is able to mine faster than the rest of the network, and he can rewrite the last end of the blockchain as he desires. This is the reason why a decentralised mining is necessary so that no one controls more than half of the mining power. But even when \(0<q<1/2\) he can try to attempt a double spend, and will succeed with a non-zero probability. The first mathematical problem consists of computing the probability that the malicious miner succeeds in rewriting the last \(n\geq 1\) blocks. We assume that the remaining relative hashrate, \(p=1-q\), consists of honest miners following the protocol rules.
+
+This problem is similar to the classical gambler's ruin problem. Nakamoto observes that the probability to catch-up \(n\) blocks is
+
+\[q_{n}=\begin{pmatrix}q\\ p\end{pmatrix}^{n}\;\;\;\text{(Nakamoto)}\] 
+
+[MISSING_PAGE_FAIL:3]
+
+validated block and the public blockchain adopts it thus loosing his reward. This type of scenario has been discussed since 2012 in bitcointalk forum, created by Nakamoto in 2010.
+
+To answer this question, we first need to develop a proper profitability model. As in any business, mining profitability is accounted by the "Profit and Loss" per unit of time. The profits of a miner come from the block rewards that include the coinbase reward in new bitcoins created, and the transaction fees of the transactions in the block. The profitability at instant \(t>0\) is given by
+
+\[\mathbf{PL}(t)=\frac{\mathbf{R}(t)-\mathbf{C}(t)}{t}\]
+
+where \(\mathbf{R}(t)\) and \(\mathbf{C}(t)\) represent, respectively, the rewards and the cost of the mining operation up to time \(t\). If we don't consider transaction fees we have
+
+\[\mathbf{R}(t)=\mathbf{N}(t)\,b\]
+
+where \(b>0\) is the coinbase reward. If we include transaction fees, the last equation remains true taking the average reward using the classical Wald Theorem.
+
+The random variable \(\mathbf{C}(t)\) representing the cost of mining operations is far more complex to determine since it depends on external factors (as electricity costs, mining hardware costs, geographic location, currency exchange rate, etc). But, fortunately, we don't need it in the comparison the profitability of different mining strategies as we explain next.
+
+The mining activity is a repetitive and the miners returns to the same initial state after some time, for instant, start mining a fresh block. A mining strategy is composed by cycles where the miner invariably returns to the initial state. It is a "game with repetition" similar to those employed by profit gamblers in casino games (when they can spot a weakness that makes the game profitable). For example, an honest miner starts a cycle each time the network, he or someone else, has validated a new block.
+
+We denote by \(\tau\) the duration of the cycle, and we are interested in _integrable_ strategies for which \(\mathbb{E}[\tau]<+\infty\) (this means that the cycles almost surely end up in finite time). Then it is easy to check, using the law of large numbers and Wald Theorem, that the long term profitability is given a.s. by the limit
+
+\[\mathbf{PL}_{\infty}=\lim_{t\rightarrow+\infty}\frac{\mathbf{R}(t)-\mathbf{C }(t)}{t}=\frac{\mathbb{E}[\mathbf{R}]-\mathbb{E}[\mathbf{C}]}{\mathbb{E}[\tau]}\]
+
+As observed before the second cost term is hard to compute, but the revenue term, that we call _revenue ratio_, is in general computable
+
+\[\mathbf{\Gamma}=\frac{\mathbb{E}[\mathbf{R}]}{\mathbb{E}[\tau]}\]
+
+For example, for an honest miner we have \(\mathbb{E}[\mathbf{R}]=p.0+q.b=qb\) and \(\mathbb{E}[\tau]=\tau_{0}\), and therefore
+
+\[\mathbf{\Gamma}_{H}=\frac{qb}{\tau_{0}}\]
+
+We have the fundamental Theorem on comparison of mining strategies with the same cost ratio. This is the case when both strategies use the full mining power at all time.
+
+**Theorem 3** ([7], 2018).: _We consider two mining strategies \(\xi\) and \(\eta\) with the same cost by unit of time. Then \(\xi\) is more profitable than \(\eta\) if and only if_
+
+\[\mathbf{\Gamma}_{\eta}\leq\mathbf{\Gamma}_{\xi}\]
+
+## 5 Protocol stability.
+
+We can now mathematically study the protocol stability. The following remarkable result (remarkable because it is hard to imagine how Nakamoto could have foreseen it) validates the proper adjustment of the protocol:
+
+**Theorem 4** ([7], 2018).: _In absence of difficulty adjustment, the optimal mining strategy is to publish immediately all mined blocks as soon as they are discovered._
+
+We remind that the difficulty of mining adjusts in about every two weeks, so at the same time we spot a weakness of the protocol that we discuss below.
+
+This Theorem holds true for any hashrate of the miner and without any assumption of the type of miners present in the network. It changes nothing that eventually there are some dishonest miners in the network.
+
+The proof is simple and is a good example of the power of martingale techniques. For a constant difficulty, the average speed of block discovery remains constant and the counting process \(\mathbf{N}(t)\) is a Poisson process with intensity \(\alpha=\frac{p}{\tau_{0}}\) where \(p\) is the relative hashrate of the miner. The cycle duration \(\tau\) is a stopping time and the revenue per cycle equals to \(\mathbf{R}=\mathbf{N}(\boldsymbol{\tau})\). Its mean value is then obtained using Doob's stopping time to the martingale \(\mathbf{N}(t)-\alpha t\). Finally we get \(\mathbf{\Gamma}\leq\mathbf{\Gamma}_{H}\).
+
+But the Bitcoin protocol does have a difficulty adjustment algorithm that is necessary, in particular during the development phase. Theorem 4 shows that this is the only vector of attack. This difficulty adjustment provides a steady monetary creation and ensures that the interblock validation time stays at around 10 minutes. A minimum delay is necessary to allow a good synchronization of all network nodes. If the hashrate accelerates without a difficulty adjustment, then the nodes will desynchronise, and many competing blockchains will appear leaving a chaotic state.
+
+## 6 Profitability of rogue strategies.
+
+In view of Theorems 3 and 4, and in order to decide if a mining strategy is more profitable than the honest strategy, we only need to compute the revenue ratio \(\mathbf{\Gamma}\) with the difficulty adjustment integrated. Selfish mining (SM strategy 1) is an example of rogue strategy. Instead of publishing a new block, the miner keeps the block secret and tries to build a longer blockchain increasing its advantage. When he makes it public, he will orphan the last mined honest blocks and will reap the rewards. To be precise, the attack cycles are defined as follows: the miner starts mining a new block on top of the official blockchain. If an honest miner finds a block first, then the cycle ends and he starts over. Otherwise, when he is the first to find a block, he keeps mining on top of it and keeping it secret. If before he mines a second block the honest network mines one public block, then he publishes his block immediately, thus trying to get a maximum proportion \(0<\gamma<1\)of honest miners adopting his block. The propagation is not instantaneous and the efficiency depends on the new parameter \(\gamma\) which represents his good connectivity to the network. A competition follows, and if the next block is mined on top of the honest block, then the selfish miner looses the rewards of this block and the attack cycle ends. If he, or his allied honest miners, mine the next block, then they publish it and the attack cycle ends again. If the attacker succeeds in mining two consecutive secret blocks at the beginning, then he continues working on his secret blockchain until he has only one block of advantage with respect to the public blockchain. In that case, he doesn't run any risk of being joined by the public blockchain and publishes all his secret blockchain, thus reaping all the rewards and ending the attack cycle again. In few words, the rogue miner spends most of his time replacing honest blocks by those that he mined in advance and kept secret. The mean duration \(\mathbb{E}[\mathbf{\tau}]\) of the attack cycle is obtained as a variation of the following result about Poisson processes.
+
+**Proposition 5** (Poisson races).: _Let \(\mathbf{N}\) and \(\mathbf{N}^{\prime}\) be two independent Poisson processes with respective parameters \(\alpha\) and \(\alpha^{\prime}\), with \(\alpha^{\prime}<\alpha\) and \(\mathbf{N}(0)=\mathbf{N}^{\prime}(0)=0\). Then, the stopping time_
+
+\[\mathbf{\sigma}=\inf[t>0;\mathbf{N}(t)=\mathbf{N}^{\prime}(t)+1]\]
+
+_is almost surely finite, and we have_
+
+\[\mathbb{E}[\mathbf{\sigma}]=\frac{1}{\alpha-\alpha^{\prime}}\;,\;\mathbb{E}[ \mathbf{N}^{\prime}(\mathbf{\sigma})]=\frac{\alpha^{\prime}}{\alpha-\alpha^{ \prime}}\;,\;\;\mathbb{E}[\mathbf{N}(\mathbf{\sigma})]=\frac{\alpha}{\alpha- \alpha^{\prime}}\;.\]
+
+The proof is a simple application of Doob's Stopping Time Theorem. Here, \(\mathbf{N}\) and \(\mathbf{N}^{\prime}\) are the counting processes of blocks mined by the honest miners and the attacker. To finish, we must compute the intensities \(\alpha\) and \(\alpha^{\prime}\). At the beginning we have \(\alpha=\alpha_{0}=\frac{p}{r_{0}}\) and \(\alpha^{\prime}=\alpha_{0}^{\prime}=\frac{q}{r_{0}}\) where \(p\) is the apparent hashrate of the honest miners and \(q\) the one of the attacker. But the existence of a selfish miner perturbs the network and slows down the production of blocks. Instead of having one block for each period \(\tau_{0}\), the progression of the official blockchain is of \(\mathbb{E}[N(\mathbf{r})\lor N^{\prime}(\mathbf{\tau})]\) blocks during \(\mathbb{E}[\mathbf{\tau}]\). After validation of 2016 official blocks, this triggers a difficulty adjustment that can be important. The new difficulty is obtained from the old one by multiplication by a factor \(\delta<1\) given by
+
+\[\delta=\frac{\mathbb{E}[N(\mathbf{\tau})\lor N^{\prime}(\mathbf{\tau})]\,\tau_{0}}{ \mathbb{E}[\mathbf{\tau}]}\]
+
+After the difficulty adjustment, the new mining parameters are \(\alpha=\alpha_{1}=\frac{a_{0}}{\delta}\) and \(\alpha^{\prime}=\alpha_{1}^{\prime}=\frac{a_{0}^{\prime}}{\delta}\). The stopping time \(\mathbf{\tau}\) and the parameter \(\delta\) can be computed using the relation \(|N(\mathbf{\tau})-N^{\prime}(\mathbf{\tau})|=1\). This can be used to compute the revenue ratio of the strategy [7]. This analysis can also checked by mining simulators.
+
+An alternative procedure consists in modelling the network by a Markov chain where the different states correspond to different degree of progress of the selfish miner. Each transition corresponds to a revenue increase \(\mathbf{\pi}\) and \(\mathbf{\pi}^{\prime}\) for the honest and selfish miner. By another application of the Law of Large Numbers we prove that the long term apparent hashrate of the strategy, defined as the proportion of mined blocks by the selfish miner compared to the total number of blocks, is given by the formula
+
+\[q^{\prime}=\frac{\mathbb{E}[\mathbf{\pi}^{\prime}]}{\mathbb{E}[\mathbf{\pi}]+\mathbb{ E}[\mathbf{\pi}^{\prime}]}\]
+
+The expectation is taken relative to the stationnary probability that exists because the Markov chain is transitive and recurrent. Indeed, the Markov chain is essentially a random walk on \(\mathbb{N}\) partially reflexive on 0. The computation of this stationnary probability proves the following Theorem:
+
+**Theorem 6** ([8], 2014).: _The apparent hashrate of the selfish miner is_
+
+\[q^{\prime}=\frac{((1+pq)(p-q)+pq)q-(1-\gamma)p^{2}q(p-q)}{p^{2}q+p-q}\]
+
+The results from [7] and [8] obtained by these different methods, are compatible. The revenue ratio \(\mathbf{\Gamma}_{1}\) and the apparent hashrate \(q^{\prime}\) are related by the following equation
+
+\[\mathbf{\Gamma}_{1}=q^{\prime}\frac{b}{\tau_{0}}\]
+
+But the first analysis is finer since it does explain the change of profitability regime after the difficulty adjustment. In particular, it allows to compute the duration before running into profitability for the attacker. The selfish miner starts by losing money, then after the difficulty adjustment that favors him, starts making profits. For example, with \(q=0.1\) and \(\gamma=0.9\), he needs to wait 10 weeks in order to be profitable. This partly explains why such an attack has never been observed in the Bitcoin network.
+
+Theorem 3 gives an explicit semi-algebraic condition on the parameters, namely \(q^{\prime}>q\), that determines the values of the parameters \(q\) and \(\gamma\) for which the selfish mining strategy is more profitable than honest mining.
+
+Theorem 4 shows that the achilles' heel of the protocole is the difficulty adjustment formula. This formula is supposed to contain the information about the total hashrate, but in reality it ignores the orphan blocks. The authors proposed a solution that incorporates this count, and this solves the stability problem of the protocol [7].
+
+There are other possible block-withholding strategies that are variations of the above strategy [9]. These are more agressive strategies. In the initial situation where the attacker succeeds to be two blocks ahead, instead of publishing the whole secret chain when he is only one block ahead, he can wait to be caught-up to release his blocks and then starts a final competition between the two competing chains. The attack cycle ends when the outcome is decided. This is the "Lead Stubborn Mining" (LSM, strategy 2). In this strategy it is important that the miner regularly publishes his secret blocks with the same height of the official blockchain, to attract part of the honest miners in order to take out hashrate from the pool of honest miners. Also in this way, even if he looses the final competition he will succeed in incorporating some of his blocks in the official blockchain and reap the corresponding rewards.
+
+Another even more aggressive variation consists in waiting not to be caught up but to be behind one block. This is the "Equal Fork Stubborn Mining Strategy" (EFSM, strategy 3). Here again, it is important to publish secret blocks regularly. Finally, the authors have considered another more agressive variation where the stubborn miner follows EFSM but then doesn't stop when he is one block behind. He keeps on mining until his delay becomes greater than a threshold \(A\) or until he successfully comes from behind, catches-up and finally takes the advantage over the official blockchain.
+
+This strategy seems desperate, because the official blockchain progress is faster, on average. But in case of catching-up the selfish miner wins the jackpot of all the blocks he replaces. This is the "A-Trailing Mining" strategy (A-TM, strategy 4). The authors of [9] conduct a numerical study of profitability by running a Montecarlo simulation and compare the profitability of the different strategies for different parameter values \((q,\gamma)\). But we can find closed form formulas for the revenue ratio of all these strategies using the precedent martingale approach.
+
+**Theorem 7**.: _([7, 10, 11, 12]) We have_
+
+\[\frac{\Gamma_{1}}{\Gamma_{H}} =\frac{(1+pq)(p-q)+pq-(1-\gamma)p^{2}(p-q)}{p^{2}q+p-q}\] \[\frac{\Gamma_{2}}{\Gamma_{H}} =\frac{p+pq-q^{2}}{p+pq-q}-\frac{p(p-q)f(\gamma,p,q)}{p+pq-q}\] \[\frac{\Gamma_{3}}{\Gamma_{H}} =\frac{1}{p}-\frac{p-q}{pq}f(\gamma,p,q)\] \[\frac{\Gamma_{4}}{\Gamma_{H}} =\frac{1+\frac{(1-\gamma)p(p-q)}{(p+pq-q^{2})(k+1)}\left(\left([ A-1]+\frac{1}{p}\frac{P_{A}(k)}{(A+1)}\right)\lambda^{2}-\frac{2}{\sqrt{1-4(1- \gamma)pq+p-q}}\right)}{\frac{p+pq-q}{p+pq-q^{2}}+\frac{(1-\gamma)pq}{p+pq-q^ {2}}(A+\lambda)\left(\frac{1}{(A+1)}-\frac{1}{A+A}\right)}\]
+
+_with_
+
+\[f(\gamma,p,q)=\frac{1-\gamma}{\gamma}\cdot\left(1-\frac{1}{2q}(1-\sqrt{1-4(1- \gamma)pq})\right)\]
+
+_and \(\lambda=q/p\), \([n]=\frac{1-\lambda^{2}}{1-\lambda}\) pour \(n\in\mathbb{N}\), \(P_{A}(\lambda)=\frac{1-\lambda A^{2}+\lambda A^{2}-\lambda A^{2}}{(1-\lambda)^ {2}}\)_
+
+We can plot the parameter regions where each strategy is the best one (Figure 3). The Catalan numbers appear naturally in the computations.
+
+\[C_{n}=\frac{1}{2n+1}\binom{2n}{n}=\frac{(2n)!}{n!(n+1)!}\;.\]
+
+For this reason their generating function appears in the formulas
+
+\[C(x)=\sum_{n=0}^{+\infty}C_{n}x^{n}=\frac{1-\sqrt{1-4x}}{2x}\]
+
+We observe that \(\sqrt{1-4pq}=p-q\) and \(C(pq)=1/p\), and this justifies the definition of new probability distributions that arise in the proofs.
+
+**Definition 8**.: A discrete random variable \(X\) taking integer values follows a Catalan distribution of the first type if we have, for \(n\geq 0\),
+
+\[\mathbb{P}[X=n]=C_{n}p(pq)^{n}\;.\]
+
+It follows a Catalan distribution of the second type if \(\mathbb{P}[X=0]=p\) and for \(n\geq 1\),
+
+\[\mathbb{P}[X=n]=C_{n-1}(pq)^{n}\;.\]
+
+It follows a Catalan distribution of the third type if \(\mathbb{P}[X=0]=p\), \(\mathbb{P}[X=1]=pq+pq^{2}\) and for \(n\geq 2\),
+
+\[\mathbb{P}[X=n]=pq^{2}C_{n-1}(pq)^{n-1}\;.\]
+
+## 7 Dyck words.
+
+We can recover this results by a direct combinatorical approach representing each attack cycle by a Dyck word.
+
+**Definition 9**.: A Dyck word is a word built from the two letter alphabet \([S,H]\) which contains as many S letters as H letters, and such that any prefix word contains more or equal S letters than H letters. We denote \(\mathcal{D}\) the set of Dyck words, and for \(n\geq 0\), \(\mathcal{D}_{n}\) the subset of Dyck worlds of length \(2n\).
+
+The relation to Catalan numbers is classical: the cardinal of \(\mathcal{D}_{n}\) is \(C_{n}\). We can encode attack cycles by a chronologic succession of block discoveries (disregarding if the blocks are made public or not). For a selfish block we use the letter S (for "selfish") and for the honest blocks the letter H (for "honest").
+
+The link between the selfish mining strategy and Dyck words is given by the following proposition:
+
+**Proposition 10**.: _The attack cycles of the SM strategy are H, SHH, SHS, and SSwH where \(w\in\mathcal{D}\)._
+
+At the end of the cycle, we can summarise and count the total number of official blocks, say \(L\), and how many of these blocks were mined by the attacker, say \(Z\). Then, for strategy 1 (SM), the random variable \(L-1\) follows a Catalan distribution of the third type, and except for some particular cases (when \(L<3\)), we always have \(L=Z\). The apparent hashrate \(q^{\prime}\) is then given by the formula:
+
+\[q^{\prime}=\frac{\mathbb{E}[Z]}{\mathbb{E}[L]}\]
+
+We can then directly recover Theorem 6 by this simpler combinatorical procedure [12]. The other rogue strategies can be studied in a similar way. The Catalan distribution of the first type arises in the study of the strategy EFSM (strategy 3), and the one of the second type for the strategy LSM (strategy 2). We can then recover all the results given by the Markov chain analysis. Unfortunately we cannot recover the more finer results obtained by martingales techniques.
+
+This sort analysis applies to other Proof-of-Work cryptocurrencies, and to Ethereum that has a more complex reward system and a different difficulty adjustment formula [13].
+
+Figure 3: Comparison of HM, SM, LSM, EFSM and A-TSM.
+
+## 8 Nakamoto double spend revisited.
+
+We come back to the fundamental double spend problem from Nakamoto Bitcoin paper discussed in section 3. In that section, we computed the probability of success of a double spend. But now, with the profitability model knowledge from section 4, we can study its profitability and get better estimates on the number of confirmations that are safe to consider a paienment definitive. The double spend strategy as presented in [1] is unsound because there is a non-zero probability of failure and in that case, if we keep mining in the hope of catching-up from far behind the official blockchain, we have a positive probability of total ruin. Also the strategy is not integrable since the expected duration of the attack is infinite. Thus, we must obviously put a threshold to the unfavorable situation where we are lagging far behind the official blockchain.
+
+We assume that the number of confirmations requested by the recipient of the transaction is \(z\) and we assume that we are never behind \(A\geq z\) blocks of the official blockchain. This defines an integrable strategy, The \(A\)-Nakamoto double spend strategy. Putting aside technical details about premining, the probability of success of this strategy is a modification of the probability from Theorem 1
+
+**Theorem 11** ([14], 2019).: _After \(z\) confirmations, the probability of success of an \(A\)-Nakamoto double spend is_
+
+\[P_{A}(z)=\frac{P(z)-\lambda^{A}}{1-\lambda^{A}}\]
+
+_where \(P(z)\) is the probability from Theorem 1 and \(\lambda=q/p\)._
+
+If \(v\) is the amount to double spend, then we can compute the revenue ratio \(\boldsymbol{\Gamma}_{A}=\mathbb{E}[\mathbf{R}]/\mathbb{E}[\boldsymbol{\tau}]\).
+
+**Theorem 12** ([14], 2019).: _With the previous notations, the expected revenue and the expected duration of the \(A\)-Nakamoto double spend strategy is_
+
+\[\mathbb{E}[\mathbf{R}_{A}]/b =\frac{qz}{2p}I_{4pq}(z,1/2)-\frac{A\lambda^{A}}{p(1-\lambda)^{2} [A]^{2}}I_{(p-q)^{2}}(1/2,z)\] \[\quad+\frac{2-\lambda+\lambda^{A+1}p^{z-1}q^{z}}{(1-\lambda)^{2} [A]}\frac{p^{z-1}q^{z}}{B(z,z)}+P_{A}(z)\left(\frac{v}{b}+1\right)\] \[\mathbb{E}[\mathbf{T}_{A}]/\tau_{0} =\frac{z}{2p}I_{4pq}(z,1/2)+\frac{A}{p(1-\lambda)^{2}[A]}I_{(p-q) ^{2}}(1/2,z)\] \[\quad-\frac{p^{z-1}q^{z}}{p(1-\lambda)\,B(z,z)}+\frac{1}{q}\]
+
+_with the notation \([n]=\frac{1-p^{z}}{1-\lambda}\) for an integer \(n\geq 0\), and \(B\) is the classical Beta function._
+
+In principle a powerful miner does not have an interest in participating in a large double spend, since doing so will undermine the foundations of his business. For a small miner with relative hashrate \(0<q<<1\) we can estimate the minimal amount of a double spend to be profitable. For this we only need to use the inequality from Theorem 3, \(\boldsymbol{\Gamma}_{A}\geq\boldsymbol{\Gamma}_{H}=qb/\tau_{0}\), and take the asymptotics \(q\to 0\) (with \(A\) and \(z\) being fixed, but the final result turns out to be independent of \(A\)).
+
+**Corollary 13**.: _When \(q\to 0\) the minimal amount \(v\) for an Nakamoto double spend with \(z\geq 1\) confirmations is_
+
+\[v\geq\frac{q^{-z}}{2\binom{z-1}{2}}b=v_{0}\;.\]
+
+For example, in practice, with a 10% hashrate, \(q=0.01\), and only one confirmation, \(z=1\), we need to double spend more than \(v_{0}/b=50\) coinbases. With the actual coinbase reward of \(b=12.5\,\lx@math@degree\) bitcoins and the actual prize over 8.300 euros, this represents more than 5 millions euros.
+
+Hence for all practical purposes and normal amount transactions, only one confirmation is enough to consider the transaction definitive.
+
+**Conclusions.** Bitcoin provides a good example of the universality of Mathematical applications and its potential to impact our society. With the glimpse we have given, we hope to have convinced our colleagues that the Bitcoin protocol also motivates some exciting Mathematics.
+
+## References
+
+* [1] S. Nakamoto, "Bitcoin: A peer-to-peer electronic cash system," _www.bitcoin.orgbitcoin.pdf_, released on November 1st 2008 on the USENET Cryptography Mailing List "Bitcoin P2P e- cash paper".
+* [2] R. Perez-Marco, "Bitcoin and decentralized trust protocols," _Newsletter European Mathematical Society_, vol. 21, no. 100, pp. 31-38, 2016.
+* [3] C. Grunspan and R. Perez-Marco, "Double spend races," _Int. Journal Theoretical and Applied Finance_, vol. 21, no. 08, 2018.
+* [4] M. Rosenfeld, "Analysis of hashrate-based double spending," _ArXiv:1402.2009_, 2014.
+* [5] C. Grunspan and R. Perez-Marco, "Satoshi risk tables," _arXiv:1702.04421_, 2017.
+* [6] E. Gloglatis and D. Zeilberger, "A combinatorial-probabilistic analysis of bitcoin attacks," _Journal of Difference Equations and its Applications_, vol. 25, no. 1, 2019.
+* [7] C. Grunspan and R. Perez-Marco, "On the profitability of selfish mining," _arXiv:1805.08281_, 2018.
+* [8] I. Eyal and E. G. Sirer, "Majority is not enough: Bitcoin mining is vulnerable," _Commun. ACM_, vol. 61, pp. 95-102, June 2018.
+* [9] K. Nayak, S. Kumar, A. K. Miller, and E. Shi, "Stubborn mining: Generalizing selfish mining and combining with an eclipse attack," _2016 IEEE European Symposium on Security and Privacy (EuroS'P)_, pp. 305-320, 2015.
+* [10] C. Grunspan and R. Perez-Marco, "On the profitability of stubborn mining," _arXiv:1808.01041_, 2018.
+* [11] C. Grunspan and R. Perez-Marco, "On the profitability of trailing mining," _arXiv:1811.09322_, 2018.
+* [12] C. Grunspan and R. Perez-Marco, "Bitcoin selfish mining and Dyck words," _arXiv:1811.09322_, 2019.
+* [13] C. Grunspan and R. Perez-Marco, "Selfish mining in Ethereum," _arXiv:1904.13330_, 2019.
+* [14] C. Grunspan and R. Perez-Marco, "On profitability of nakamoto double spend," _arXiv:1912.06412_, 2019.

2003.00002v1.mmd

@@ -0,0 +1,389 @@
+# A nonlinear extension of Korovkin's theorem
+
+Sorin G. Gal
+
+Department of Mathematics, University of California, Berkeley, CA 94720, USA sgal@math.berkeley.edu
+
+Constantin P. Niculescu
+
+Department of Mathematics, University of California, Berkeley, CA 94720, USA pniculescu@math.berkeley.edu Dedicated to Professor Nicolae Dinculeanu, on the occasion of his 95th birthday.
+
+November 6, 2021
+
+###### Abstract.
+
+In this paper we extend the classical Korovkin theorems to the framework of comonotone additive, sublinear and monotone operators. Based on the theory of Choquet capacities, several concrete examples illustrating our results are also discussed.
+
+Key words and phrases:Choquet integral, Korovkin theorem, comonotone additivity, monotone operator, sublinear operator 2000 Mathematics Subject Classification: 41A35, 41A36, 41A63
+
+## 1. Introduction
+
+One of the most elegant results in the theory of approximation is Korovkin's theorem, that provides a generalization of the well-known proof of Weierstrass's classical approximation theorem as was given by Bernstein.
+
+**Theorem 1**.: (Korovkin [18], [19]) _Let \((L_{n})_{n}\) be a sequence of positive linear operators that map \(C([0,1])\) into itself. Suppose that the sequence \((L_{n}(f))_{n}\) converges to \(f\) uniformly on \([0,1]\) for each of the test functions \(1,\,x\) and \(x^{2}\). Then this sequence converges to \(f\) uniformly on \([0,1]\) for every \(f\in C([0,1])\)._
+
+Simple examples show that the assumption concerning the positivity of the operators \(L_{n}\) cannot be dropped. What about the assumption on their linearity?
+
+Over the years, many generalizations of Theorem 1 appeared, in a variety of settings including important Banach function spaces. A nice account on the present state of art is offered by the authoritative monograph of Altomare and Campiti [3] and the excellent survey of Altomare [2]. The literature concerning the subject of Korovkin type theorems is really huge, a search on Google offering more than 26,000 results. However, except for Theorem 2.7 in the 1973 paper of Bauer [4], the extension of this theory beyond the framework of linear functional analysis remained largely unexplored.
+
+Inspired by the Choquet theory of integrability with respect to a nonadditive measure, we will prove in this paper that the restriction to the class of positive linear operators can be relaxed by considering operators that verify a mix of conditions characteristic for Choquet's integral.
+
+As usually, for \(X\) a Hausdorff topological space we will denote by \(\mathcal{F}(X)\) the vector lattice of all real-valued functions defined on \(X\) endowed with the pointwise ordering. Two important vector sublattices of it are
+
+\[C(X)=\{f\in\mathcal{F}(X):\ f\text{ continuous}\}\]and
+
+\[C_{b}(X)=\left\{f\in\mathcal{F}(X):\text{ $f$ continuous and bounded}\right\}.\]
+
+With respect to the the sup norm, \(C_{b}(X)\) becomes a Banach lattice. See [22] for the theory of these spaces.
+
+Suppose that \(X\) and \(Y\) are two Hausdorff topological spaces and \(E\) and \(F\) are respectively vector sublattices of \(C(X)\) and \(C(Y).\) An operator \(T:E\to F\) is called:
+
+- _sublinear_ if it is both _subadditive_, that is,
+
+\[T(f+g)\leq T(f)+T(g)\quad\text{for all $f,g\in E$},\]
+
+and _positively homogeneous,_ that is,
+
+\[T(af)=aT(f)\quad\text{for all $a\geq 0$ and $f\in E$};\]
+
+- _monotonic_ if \(f\leq g\) in \(E\) implies \(T(f)\leq T(g);\)
+
+- _comonotonic additive_ if \(T(f+g)=T(f)+T(g)\) whenever the functions \(f,g\in E\) are comonotone in the sense that
+
+\[\left(f(s)-f(t)\right)\cdot\left(g(s)-g(t)\right)\geq 0\text{ for all $s,t\in X$}.\]
+
+Our main result extends Korovkin's results to the framework of operators acting on vector lattices of functions of several variables that play the properties of sublinearity, monotonicity and comonotonic additivity. We use families of test functions including the canonical projections on \(\mathbb{R}^{N},\)
+
+\[\text{pr}_{k}:(x_{1},...,x_{N})\to x_{k},\quad k=1,...,N.\]
+
+**Theorem 2**.: (The nonlinear extension of Korovkin's theorem: the several variables case) _Suppose that \(X\) is a locally compact subset of the Euclidean space \(\mathbb{R}^{N}\) and \(E\) is a vector sublattice of \(\mathcal{F}(X)\) that contains the test functions \(1,\ \pm\text{pr}_{1},...,\ \pm\text{pr}_{N}\) and \(\sum_{k=1}^{N}\text{pr}_{k}^{2}.\)_
+
+\((i)\) _If \((T_{n})_{n}\) is a sequence of monotone and sublinear operators from \(E\) into \(E\) such that_
+
+\[\lim_{n\to\infty}T_{n}(f)=f\quad\text{uniformly on the compact subsets of $X$} \tag{1.1}\]
+
+_for each of the \(2N+2\) aforementioned test functions, then the property (1.1) also holds for all nonnegative functions \(f\) in \(E\cap C_{b}(X).\)_
+
+\((ii)\) _If, in addition, each operator \(T_{n}\) is comonotone additive, then \((T_{n}(f))_{n}\) converges to \(f\) uniformly on the compact subsets of \(X\), for every \(f\in E\cap C_{b}\left(X\right).\)_
+
+_Notice that in both cases \((i)\) and \((ii)\) the family of testing functions can be reduced to \(1,\ -\text{pr}_{1},...,\ -\text{pr}_{N}\) and \(\sum_{k=1}^{N}\text{pr}_{k}^{2}\) when \(K\) is included in the positive cone of \(\mathbb{R}^{N}\). Also, the convergence of \((T_{n}(f))_{n}\) to \(f\) is uniform on \(X\) when \(f\in E\) is uniformly continuous and bounded on \(X.\)_
+
+The details of this result make the objective of Section 2.
+
+Theorem 2 extends not only Korovkin's original result (which represents the particular case where \(N=1,\)\(K=[0,1],\) all operators \(T_{n}\) are linear bounded and monotone, and the function \(\text{pr}_{1}\) is the identity of \(K\)) but also the several variable version of it due to due to Volkov [25]. It encompasses also the technique of smoothing kernels, in particular Weierstrass' argument for the Weierstrass approximation theorem: for every bounded uniformly continuous function \(f:\mathbb{R}\to\mathbb{R},\)
+
+\[\left(W_{h}f\right)(t)=\frac{1}{h\sqrt{\pi}}\int_{-\infty}^{\infty}f(s)e^{-( s-t)^{2}/h^{2}}ds\longrightarrow f(t)\]uniformly on \(\mathbb{R}\) as \(h\to 0.\)
+
+Applications of Theorem 2 in the nonlinear setting are presented in Section 3. They are all based on Choquet's theory on integration with respect to a capacity. Indeed, this theory, which was initiated by Choquet [6], [7] in the early 1950s, represents a major source of comonotonic additive, sublinear and monotone operators.
+
+It is worth mentioning that nowadays Choquet's theory provides powerful tools in decision making under risk and uncertainty, game theory, ergodic theory, pattern recognition, interpolation theory and very recently on transport under uncertainty. See Adams [1], Denneberg [8], Follmer and Schied [9], Wang and Klir [26], Wang and Yan [27], Gal and Niculescu [13] as well as the references therein.
+
+For the convenience of reader we summarized in the Appendix at the end of this paper some basic facts concerning this theory.
+
+Some nonlinear extension of Korovkin's theorem within the framework of compact spaces are presented in Section 4.
+
+## 2. Proof of Theorem 2
+
+Before to detail the proof of Theorem 2 some preliminary remarks on the behavior of operators \(T:C_{b}(X)\to C_{b}(Y)\) are necessary.
+
+If \(T\) is subadditive and monotone, then it verifies the inequality
+
+\[\left|T(f)-T(g)\right|\leq T\left(\left|f-g\right|\right)\quad\text{for all }f,g. \tag{2.1}\]
+
+Indeed, \(f\leq g+\left|f-g\right|\) yields \(T(f)\leq T(g)+T\left(\left|f-g\right|\right),\) i.e., \(T(f)-T(g)\leq T\left(\left|f-g\right|\right)\), and interchanging the role of \(f\) and \(g\) we infer that \(-\left(T(f)-T(g)\right)\leq T\left(\left|f-g\right|\right).\)
+
+If \(T\) is linear, then the property of monotonicity is equivalent to that of positivity, whose meaning is
+
+\[T(f)\geq 0\quad\text{for all }f\geq 0.\]
+
+If the operator \(T\) is monotone and positively homogeneous then necessarily
+
+\[T(0)=0.\]
+
+Every positively homogeneous and comonotonic additive operator \(T\) verifies the formula
+
+\[T(f+a\cdot 1)=T(f)+aT(1)\quad\text{for all }f\text{ and all }a\in[0,\infty); \tag{2.2}\]
+
+indeed, \(f\) is comonotonic to any constant function.
+
+Proof of Theorem 2.: \((i)\) In order to make more easier the handling of the test functions we denote
+
+\[e_{0}=1,\ e_{k}=\operatorname{pr}_{k}\ \left(k=1,...N\right)\text{ and }e_{N+1}=\sum_{k=1}^{N} \operatorname{pr}_{k}^{2}.\]
+
+Replacing each operator \(T_{n}\) by \(T_{n}/T_{n}(e_{0}),\) we may assume that \(T_{n}(e_{0})=1\) for all \(n.\)
+
+Let \(f\in E\cap C_{b}(\Omega)\) and let \(K\) be a compact subset of \(X.\) Then for every \(\varepsilon>0\) there is \(\bar{\delta}>0\) such that
+
+\[\left|f(s)-f(t)\right|\leq\varepsilon\quad\text{for every }t\in K\text{ and }s\in X\text{ with }\|s-t\|\leq\bar{\delta};\]
+
+this can be easily proved by reductio ad absurdum.
+
+If \(\|s-t\|\geq\tilde{\delta}\), then
+
+\[|f(s)-f(t)|\leq\frac{2\|f\|_{\infty}}{\tilde{\delta}^{2}}\cdot\|s-t\|^{2},\]
+
+so that letting \(\delta=2\|f\|_{\infty}/\tilde{\delta}^{2}\) we obtain the estimate
+
+\[|f(s)-f(t)|\leq\varepsilon+\delta\cdot\|s-t\|^{2} \tag{2.3}\]
+
+for all \(t\in K\) and \(s\in X.\) Since \(K\) is a compact set, it can embedded into an \(N\)-dimensional cube \([a,b]^{N}\) for suitable \(b\geq 0\geq a\) and the estimate (2.3) yields
+
+\[|f(s)-f(t)e_{0}|\leq\varepsilon e_{0}\\ +\delta(\varepsilon)\left[e_{N+1}^{2}(s)+2\sum_{k=1}^{N}\left(e_{ k}(t)-a\right)\left(-e_{k}(s)\right)\right.\\ \left.-2a\sum_{k=1}^{N}e_{k}(s)+\left\|t\right\|^{2}e_{0}(s)\right].\]
+
+Taking into account the formula (2.1) and the fact that the operators \(T_{n}\) are subadditive and positively homogeneous, we infer that
+
+\[|T_{n}(f)(s)-f(t)|=|T_{n}(f)(s)-T_{n}(f(t)e_{0})(s)|\leq T_{n} \left(|f(s)-f(t)e_{0}|\right)\\ \leq\varepsilon+\delta(\varepsilon)\left[T_{n}(e_{N+1}^{2})(s)+2 \sum_{k=1}^{N}\left(e_{k}(t)-a\right)T_{n}(-e_{k})(s)\right.\\ \left.-2a\sum_{k=1}^{N}T_{n}\left(e_{k}\right)(s)\right)+\left\| t\right\|^{2}\right]\]
+
+for every \(n\in\mathbb{N}\) and \(s,t\in K.\) Here we used the assumption that \(f\) is nonnegative. By our hypothesis,
+
+\[T_{n}(e_{N+1}^{2})(s)+2\sum_{k=1}^{N}\left(e_{k}(s)-a\right)T_{n}(-e_{k})(s)- 2a\sum_{k=1}^{N}T_{n}\left(e_{k}\right)(s))+\left\|s\right\|^{2}\to 0\]
+
+uniformly on \(K\) as \(n\to\infty.\) Therefore
+
+\[\limsup_{n\to\infty}|T_{n}(f)(s)-f(s)|\leq\varepsilon\]
+
+whence we conclude that \(T_{n}(f)\to f\) uniformly on \(K\) because \(\varepsilon\) was arbitrarily fixed.
+
+\((ii)\) Suppose in addition that each operator \(T_{n}\) is also comonotone additive. According to the assertion \((i)\),
+
+\[T_{n}(f+\|f\|e_{0})\to f+\|f\|e_{0},\quad\text{uniformly on }K.\]
+
+Since a constant function is comonotone with any arbitrary function, using the comonotone additivity of \(T_{n}\) it follows that \(T_{n}(f+\|f\|e_{0})=T_{n}(f)+\|f\|\cdot T_{n}(e_{0}).\) Therefore \(T_{n}(f)\to f\) uniformly on \(K\)When \(K\) is included in the positive cone of \(\mathbb{R}^{N},\) it can embedded into an \(N\)-dimensional cube \([0,b]^{N}\) for a suitable \(b>0\) and the estimate (2.3) yields
+
+\[|f(s)-f(t)e_{0}|\leq\varepsilon e_{0}\\ +\delta(\varepsilon)\left[e_{N+1}^{2}(s)+2\sum_{k=1}^{N}e_{k}(t) \left(-e_{k}(s)\right)+\left\|t\right\|^{2}e_{0}(s)\right].\]
+
+Proceeding as above, we infer that
+
+\[|T_{n}(f)(s)-f(t)|\\ \leq\varepsilon+\delta(\varepsilon)\left[T_{n}(e_{N+1}^{2})(s)+ 2\sum_{k=1}^{N}e_{k}(t)T_{n}(-e_{k})(s)+\left\|t\right\|^{2}\right]\]
+
+for every \(n\in\mathbb{N}\) and \(s,t\in K,\) provided that \(f\geq 0.\) As a consequence, in both cases \((i)\) and \((ii)\) the family of testing functions can be reduced to \(e_{0},-e_{1},...,-e_{N}\) and \(e_{N+1}.\)
+
+When dealing with functions \(f\in E\) uniformly continuous and bounded on \(X,\) an inspection of the argument above shows that \(f\) verifies an estimate of the form (2.3) for all \(s,t\in X,\) a fact that implies the convergence of \((T_{n}(f))_{n}\) to \(f\) uniformly on \(X.\)
+
+## 3. Applications of Theorem 2
+
+We will next discuss several examples of operators illustrating Theorem 2. They are all based on Choquet's theory of integration with respect to a capacity \(\mu,\) in our case the restriction of the monotone and submodular capacity
+
+\[\mu(A)=(\mathcal{L}(A))^{1/2}\]
+
+to various compact subintervals of \(\mathbb{R};\) here \(\mathcal{L}\) denotes the Lebesgue measure on real line. The necessary background on Choquet's theory is provided by the Appendix at the end of this paper.
+
+The \(1\)-dimensional case of Theorem 2 is illustrated by the following three families of nonlinear operators, first considered in [11]:
+
+- the _Bernstein-Kantorovich-Choquet_ operators act on \(C([0,1])\) by the formula
+
+\[K_{n,\mu}(f)(x)=\sum_{k=0}^{n}\frac{(C)\int_{k/(n+1)}^{(k+1)/(n+1)}f(t)d\mu}{ \mu([k/(n+1),(k+1)/(n+1)])}\cdot\binom{n}{k}x^{k}(1-x)^{n-k};\]
+
+- the _Szasz-Mirakjan-Kantorovich-Choquet_ operators act on \(C([0,\infty))\) by the formula
+
+\[S_{n,\mu}(f)(x)=e^{-nx}\sum_{k=0}^{\infty}\frac{(C)\int_{k/n}^{(k+1)/n}f(t)d \mu}{\mu([k/n,(k+1)/n])}\cdot\frac{(nx)^{k}}{k!};\]
+
+- the _Baskakov-Kantorovich-Choquet_ operators act on \(C([0,\infty))\) by the formula
+
+\[V_{n,\mu}(f)(x)=\sum_{k=0}^{\infty}\frac{(C)\int_{k/n}^{(k+1)/n}f(t)d\mu}{ \mu([k/n,(k+1)/n])}\cdot\binom{n+k-1}{k}\frac{x^{k}}{(1+x)^{n+k}}.\]
+
+Since the Choquet integral with respect to a submodular capacity \(\mu\) is comonotone additive, sublinear and monotone, it follows that all above operators also have these properties.
+
+Clearly, \(K_{n,\mu}(e_{0})(x)=1\) and by Corollary 3.6\((i)\) in [11] we immediately get that \(K_{n,\mu}(e_{2})(x)\to e_{2}(x)\) uniformly on \([0,1]\). Again by Corollary 3.6\((i)\), it follows that \(K_{n,\mu}(1-e_{1})(x)\to 1-e_{1}\), uniformly on \([0,1]\). Since \(K_{n,\mu}\) is comonotone additive,
+
+\[K_{n,\mu}(1-e_{1})(x)=K_{n,\mu}(e_{0})(x)+K_{n,\mu}(-e_{1})(x),\]
+
+which implies that \(K_{n,\mu}(-e_{1})\to-e_{1}\) uniformly on \([0,1].\) Therefore the operators \(K_{n,\mu}\) satisfy the hypothesis of Theorem 2, whence the conclusion
+
+\[K_{n,\mu}(f)(x)\to f(x)\text{ uniformly for every }f\in C([0,1]).\]
+
+Similarly, one can show that the operators \(S_{n,\mu}\) and \(V_{n,\mu}\) satisfy the hypothesis of Theorem 2 for \(N=1\) and \(X=[0,+\infty)\). In the first case, notice that the condition \(S_{n,\mu}(e_{0})=e_{0}\) is trivial. The convergence of the sequence of functions \(S_{n,\mu}(e_{2})(x)\) will be settled by computing the integrals \(\sqrt{n}\cdot(C)\int_{k/n}^{(k+1)/n}t^{2}d\mu\). We have
+
+\[\sqrt{n}\cdot(C)\int_{k/n}^{(k+1)/n}t^{2}d\mu=\sqrt{n}\int_{0}^{ \infty}\mu(\{t\in[k/n,(k+1)/n]:t\geq\sqrt{\alpha}\})d\alpha\\ =\sqrt{n}\int_{0}^{((k+1)/n)^{2}}\mu(\{t\in[k/n,(k+1)/n]:t\geq \sqrt{\alpha}\})d\alpha\\ =\sqrt{n}\int_{0}^{(k/n)^{2}}\mu(\{t\in[k/n,(k+1)/n]:t\geq\sqrt{ \alpha}\})d\alpha\\ +\sqrt{n}\int_{(k/n)^{2}}^{((k+1)/n)^{2}}\mu(\{t\in[k/n,(k+1)/n]: t\geq\sqrt{\alpha}\})d\alpha\\ =\sqrt{n}\cdot\left(\frac{k}{n}\right)^{2}\cdot\frac{1}{\sqrt{n}} +\sqrt{n}\cdot\int_{(k/n)^{2}}^{(((k+1)/n)^{2}}\sqrt{(k+1)/n-\sqrt{\alpha}}d\alpha \\ =\left(\frac{k}{n}\right)^{2}+\sqrt{n}\cdot\int_{0}^{1/n}\beta^{1 /2}((k+1)/n-\beta)d\beta\\ =\left(\frac{k}{n}\right)^{2}+\sqrt{n}\cdot\frac{2(k+1)}{n}\cdot \frac{2}{3}\cdot\beta^{3/2}|_{0}^{1/n}-2\sqrt{n}\cdot\frac{2}{5}\beta^{5/2}|_{ 0}^{1/n}\\ =\frac{1}{15n^{2}}\left(15k^{2}+20k+8\right).\]
+
+This immediately implies
+
+\[S_{n,\mu}(e_{2})(x)=S_{n}(e_{2})(x)+\frac{4}{3n}S_{n}(e_{1})(x)+\frac{4}{3n^{2 }}-\frac{4}{5n^{2}}\to e_{2}(x),\]
+
+uniformly on every compact subinterval \([0,a]\). Here \(S_{n}\) denotes the classical Szasz-Mirakjan-Kantorovich operator, associated to the Lebesgue measure.
+
+It remains to show that \(S_{n,\mu}(-e_{1})(x)\to-e_{1}(x)\), uniformly on every compact subinterval \([0,a]\). For this goal we have to perform the following computation:
+
+\[\sqrt{n}\cdot(C)\int_{k/n}^{(k+1)/n}(-t)d\mu=\int_{-\infty}^{0} \left\{\mu(\{\omega\in[k/n,(k+1)/n]:-\omega\geq\alpha\})-\frac{1}{\sqrt{n}} \right\}d\alpha\\ =\sqrt{n}\int_{-k/n}^{0}\left\{\mu(\{\omega\in[k/n,(k+1)/n]:\omega \leq-\alpha\})-\frac{1}{\sqrt{n}}\right\}d\alpha\\ +\sqrt{n}\int_{-(k+1)/n}^{-k/n}\left\{\mu(\{\omega\in[k/n,(k+1)/n ]:\omega\leq-\alpha\})-\frac{1}{\sqrt{n}}\right\}d\alpha\\ =-\frac{k}{n}+\sqrt{n}\cdot\int_{-(k+1)/n}^{-k/n}\left(\sqrt{- \alpha-k/n}-\frac{1}{\sqrt{n}}\right)d\alpha\\ =-\frac{k}{n}+\sqrt{n}\int_{k/n}^{(k+1)/n}\sqrt{\beta-k/n}d\beta -\frac{1}{n}\\ =-\frac{k}{n}+\sqrt{n}\int_{0}^{1/n}\beta^{1/2}d\beta-\frac{1}{n }=-\frac{3k+1}{3n}.\]
+
+Consequently
+
+\[S_{n,\mu}(-e_{1})(x)=S_{n}(-e_{1})(x)-\frac{1}{n}\to-x,\]
+
+uniformly on any compact interval \([0,a]\).
+
+In a similar way, one can be prove that the Baskakov-Kantorovich-Choquet operators \(V_{n,\mu}\) satisfy the hypothesis of Theorem 2.
+
+The several variables framework can be illustrated by the following special type of Bernstein-Durrmeyer-Choquet operators (see [14] for the general case) that act on the space of continuous functions defined on the \(N\)-simplex
+
+\[\Delta_{N}=\{(x_{1},...,x_{N}):0\leq x_{1},...,x_{N}\leq 1,\,0\leq x_{1}+\cdots+x _{N}\leq 1\}\]
+
+via the formulas
+
+\[M_{n,\mu}(f)(\mathbf{x})=B_{n}(f)(\mathbf{x})-f(\mathbf{x})+x_{N}^{n}\left[ \frac{(C)\int_{\Delta_{N}}f(t_{1},...t_{N})t_{N}^{n}d\mu}{(C)\int_{\Delta_{N} }t_{N}^{n}d\mu}-f(0,...,0,1)\right].\]
+
+Here \(\mathbf{x}=(x_{1},...,x_{N})\), \(B_{n}(f)(\mathbf{x})\) is the multivariate Bernstein polynomial and \(\mu=\sqrt{\mathcal{L}_{N}}\), where \(\mathcal{L}_{N}\) is the \(N\)-dimensional Lebesgue measure. The fact that these operators verify the hypotheses of Theorem 2 is an exercise left to the reader.
+
+## 4. The case of spaces of functions defined on compact spaces
+
+The alert reader has probably already noticed that the basic clue in the proof of Theorem 2 is the estimate (2.3), characterized in [21] (see also [20]) as a property of absolute continuity. This estimate occurs in the larger context of spaces \(C(M)\), where \(M\) is a metric space on which is defined a _separating function_, that is, a nonnegative continuous function \(\gamma:M\times M\to\mathbb{R}\) such that
+
+\[\gamma(s,t)=0\text{ implies }s=t.\]
+
+If \(M\) is a compact subset of \(\mathbb{R}^{N}\), and \(f_{1},...,f_{m}\in C(M)\) is a family of functions which separates the points of \(M\) (in particular this is the case of the coordinatefunctions \(\mathrm{pr}_{1},...,\mathrm{pr}_{N}\)), then
+
+\[\gamma(s,t)=\sum_{k=1}^{m}\left(f_{k}(s)-f_{k}(t)\right)^{2} \tag{4.1}\]
+
+is a separating function.
+
+**Lemma 1**.: (_See [21]_) _If \(K\) is a compact metric space, and \(\gamma:K\times K\rightarrow\mathbb{R}\) is a separating function, then any real-valued continuous function \(f\) defined on \(K\) verifies an estimate of the following form_
+
+\[|f(s)-f(t)|\leq\varepsilon+\delta(\varepsilon)\gamma(s,t)\quad\text{for all }s,t\in K.\]
+
+The separating functions play an important role in obtaining Korovkin-type theorems. A sample is as follows:
+
+**Theorem 3**.: _Suppose that \(K\) is a compact metric space and \(\gamma\) is a separating function for \(M.\) If \(T_{n}:C(K)\to C(K)\)\((n\in\mathbb{N})\) is a sequence of comonotone additive, sublinear and monotone operators such that \(T_{n}(1)\to 1\) uniformly and_
+
+\[T_{n}(\gamma(\cdot,t))(t)\to 0\quad\text{uniformly in }t, \tag{4.2}\]
+
+_then \(T_{n}(f)\to f\) uniformly for each \(f\in C(K).\)_
+
+The details are similar to that used for Theorem 2, so they will be omitted.
+
+In a similar way one can prove the following nonlinear extension of the Korovkin type theorem (due in the linear case to Schempp [23] and Grossman [17]):
+
+**Theorem 4**.: _Let \(X\) be a compact Hausdorff space and \(\mathcal{F}\) a subset of \(C(X)\) that separates the points of \(X\). If \((T_{n})_{n}\) is a sequence of comonotonic additive, sublinear and monotone operators that map \(C(X)\) into \(C(X)\) and satisfy the conditions \(\lim_{n\rightarrow\infty}T_{n}(f^{k})=f^{k}\) for each \(f\) in \(\mathcal{F}\) and \(k=0,1,2,\) then_
+
+\[\lim_{n\rightarrow\infty}T_{n}(f)=f,\]
+
+_for every \(f\) in \(C(X)\)._
+
+## 5. Appendix. Some basic facts on capacities and Choquet integral
+
+For the convenience of the reader we will briefly recall in this section some basic facts concerning the mathematical concept of capacity and the integral associated to it. Full details are to be found in the books of Denneberg [8], Grabisch [16] and Wang and Klir [26].
+
+Let \((X,\mathcal{A})\) be an arbitrarily fixed measurable space, consisting of a nonempty abstract set \(X\) and a \(\sigma\)-algebra \(\mathcal{A}\) of subsets of \(X.\)
+
+**Definition 1**.: _A set function \(\mu:\mathcal{A}\rightarrow[0,\infty)\) is called a capacity if \(\mu(\emptyset)=0\) and_
+
+\[\mu(A)\leq\mu(B)\quad\text{for all }A,B\in\mathcal{A},\text{ with }A\subset B\]
+
+_A capacity is called normalized if \(\mu(X)=1;\)_
+
+An important class of normalized capacities is that of probability measures (that is, the capacities playing the property of \(\sigma\)-additivity). Probability distortions represents a major source of nonadditive capacities. Technically, one start with a probability measure \(P:\mathcal{A}\rightarrow]0,1]\) and applies to it a distortion \(u:[0,1]\rightarrow[0,1],\) that is, a nondecreasing and continuous function such that \(u(0)=0\) and \(u(1)=1;\)for example, one may chose \(u(t)=t^{a}\) with \(\alpha>0.\)The _distorted probability_\(\mu=u(P)\) is a capacity with the remarkable property of being continuous by descending sequences, that is,
+
+\[\lim_{n\to\infty}\mu(A_{n})=\mu\left(\bigcap_{n=1}^{\infty}A_{n}\right)\]
+
+for every nonincreasing sequence \((A_{n})_{n}\) of sets in \(\mathcal{A}.\) Upper continuity of a capacity is a generalization of countable additivity of an additive measure. Indeed, if \(\mu\) is an additive capacity, then upper continuity is the same with countable additivity. When the distortion \(u\) is concave (for example, when \(u(t)=t^{a}\) with \(0<\alpha<1\)), then \(\mu\) is also _submodular_ in the sense that
+
+\[\mu(A\cup B)+\mu(A\cap B)\leq\mu(A)+\mu(B)\quad\text{for all }A,B\in\mathcal{A}.\]
+
+Another simple technique of constructing normalized submodular capacities \(\mu\) on a measurable space \((X,\mathcal{A})\) is by allocating to it a probability space \((Y,\mathcal{B},P)\) via a map \(\rho:\mathcal{A}\to\mathcal{B}\) such that
+
+\[\rho(\emptyset)=\emptyset,\ \rho(X)=Y\text{ and }\] \[\rho\left(\bigcap_{n=1}^{\infty}A_{n}\right)=\bigcap_{n=1}^{ \infty}\rho(A_{n})\quad\text{for every sequence of sets }A_{n}\in\mathcal{A}.\]
+
+This allows us to define \(\mu\) by the formula
+
+\[\mu(A)=1-P\left(\rho(X\backslash A)\right).\]
+
+See Shafer [24] for details.
+
+The next concept of integrability with respect to a capacity refers to the whole class of random variables, that is, to all functions \(f:X\to\mathbb{R}\) such that \(f^{-1}(A)\in\mathcal{A}\) for every Borel subset \(A\) of \(\mathbb{R}\).
+
+**Definition 2**.: _The Choquet integral of a random variable \(f\) with respect to the capacity \(\mu\) is defined as the sum of two Riemann improper integrals,_
+
+\[(C)\int_{X}fd\mu =\int_{0}^{+\infty}\mu\left(\left\{x\in X:f(x)\geq t\right\} \right)dt\] \[+\int_{-\infty}^{0}\left[\mu\left(\left\{x\in X:f(x)\geq t\right\} \right)-\mu(X)\right]dt,\]
+
+_Accordingly, \(f\) is said to be Choquet integrable if both integrals above are finite._
+
+If \(f\geq 0\), then the last integral in the formula appearing in Definition 2 is \(0\).
+
+The inequality sign \(\geq\) in the above two integrands can be replaced by \(>\); see [26], Theorem 11.1, p. 226.
+
+Every bounded random variable is Choquet integrable. The Choquet integral coincides with the Lebesgue integral when the underlying set function \(\mu\) is a \(\sigma\)-additive measure.
+
+The integral of a function \(f:X\to\mathbb{R}\) on a set \(A\in\mathcal{A}\) is defined by the formula
+
+\[(C)\int_{A}fd\mu=(C)\int_{X}fd\mu_{A}\]
+
+where \(\mu_{A}\) is the capacity defined by \(\mu_{A}(B)=\mu(B\cap A)\) for all \(B\in\mathcal{A}\).
+
+We next summarize some basic properties of the Choquet integral.
+
+**Remark 1**.: \((a)\) _If \(\mu:\mathcal{A}\rightarrow[0,\infty)\) is a capacity, then the associated Choquet integral is a functional on the space of all bounded random variables such that:_
+
+\[f\geq 0\text{ implies }(C)\int_{A}fd\mu\geq 0\ \ \ \ (\text{positivity})\]
+
+\[f\leq g\text{ implies }\left(C\right)\int_{A}fd\mu\leq(C)\int_{A}gd\mu\ \ \ \ (\text{monotonicity})\]
+
+\[(C)\int_{A}afd\mu=a\cdot\left((C)\int_{A}fd\mu\right)\text{ for }a\geq 0\ \ \ \ \ (\text{positive homogeneity})\]
+
+\[(C)\int_{A}1\cdot d\mu(t)=\mu(A)\ \ \ (\text{calibration});\]
+
+_see [8], Proposition 5.1\((ii)\), p. 64, for a proof of the property of positive homogeneity._
+
+\((b)\) _In general, the Choquet integral is not additive but, if the bounded random variables \(f\) and \(g\) are comonotonic, then_
+
+\[(C)\int_{A}(f+g)d\mu=(C)\int_{A}fd\mu+(\text{Ch})\int_{A}gd\mu.\]
+
+_This is usually referred to as the property of comonotonic additivity and was first noticed by Delacherie [10]. An immediate consequence is the property of translation invariance,_
+
+\[(C)\int_{A}(f+c)d\mu=(C)\int_{A}fd\mu+c\cdot\mu(A)\]
+
+_for all \(c\in\mathbb{R}\) and all bounded random variables \(f.\) For details, see [8], Proposition 5.1, \((vi)\), p. 65._
+
+\((c)\) _If \(\mu\) is an upper continuous capacity, then the Choquet integral is upper continuous in the sense that_
+
+\[\lim_{n\rightarrow\infty}\left((C)\int_{A}f_{n}d\mu\right)=(C)\int_{A}fd\mu\]
+
+_whenever \((f_{n})_{n}\) is a nonincreasing sequence of bounded random variables that converges pointwise to the bounded variable \(f.\) This is a consequence of the Bepo Levi monotone convergence theorem from the theory of Lebesgue integral._
+
+\((d)\) _Suppose that \(\mu\) is a submodular capacity. Then the associated Choquet integral is a subadditive functional, that is,_
+
+\[(C)\int_{A}(f+g)d\mu\leq(C)\int_{A}fd\mu+(C)\int_{A}gd\mu\]
+
+_for all bounded random variables \(f\) and \(g.\) See [8], Corollary 6.4, p. 78. and Corollary 13.4, p. 161. It is also a submodular functional in the sense that_
+
+\[(C)\int_{A}\sup\left\{f,g\right\}d\mu+(C)\int_{A}\inf\{f,g\}d\mu\leq(C)\int_{A }fd\mu+(C)\int_{A}gd\mu\]
+
+_for all bounded random variables \(f\) and \(g.\) See [5], Theorem 13\((c).\)_
+
+A characterization of Choquet integral in terms of additivity on comonotonic functions is provided by the following analogue of the Riesz representation theorem. See Zhou [28], Theorem 1 and Lemma 3, for a simple (and more general) argument.
+
+**Theorem 5**.: _Suppose that \(I:C(X)\to\mathbb{R}\) is a comonotonically additive and monotone functional with \(I(1)=1\). Then it is also upper continuous and there exists a unique upper continuous normalized capacity \(\mu:\mathcal{B}(X)\to[0,1]\) such that \(I\) coincides with the Choquet integral associated to it._
+
+_On the other hand, according to Remark 1, the Choquet integral associated to any upper continuous capacity is a comonotonically additive, monotone and upper continuous functional._
+
+Notice that under the assumptions of Theorem 5, the capacity \(\mu\) is submodular if and only if the functional \(I\) is submodular.
+
+## References
+
+* [1] Adams, D.R.: Choquet integrals in potential theory. Publ. Mat. **42**, 3-66 (1998)
+* [2] Altomare, F.: Korovkin-type theorems and positive operators. Surveys in Approximation Theory **6**, 92-164 (2010)
+* [3] Altomare, F., Campiti, M.: Korovkin-Type Approximation Theory and Its Applications. de Gruyter Studies in Mathematics, Vol. **17**, de Gruyter, Berlin (1994, reprinted 2011)
+* [4] Bauer, H.: Theorems of Korovkin type for adapted spaces. Annales de l'institut Fourier **23**(4), 245-260 (1973)
+* [5] Cerreira-Vioglio, S., Maccheroni, F., Marinacci, M., Montrucchio, L.: Signed integral representations of comonotonic additive functionals. J. Math. Anal. Appl. **385**, 895-912 (2012)
+* [6] Choquet, G.: Theory of capacities. Annales de l' Institut Fourier **5**, 131-295 (1954)
+* [7] Choquet, G.: La naissance de la theorie des capacites: reflexion sur une experience personnelelle. Comptes rendus de l'Academie des sciences, Serie generale, La Vie des sciences **3**, 385-397 (1986)
+* [8] Denneberg, D.: Non-Additive Measure and Integral. Kluwer Academic Publisher, Dordrecht (1994)
+* [9] Follmer, H., Schied, A.: Stochastic Finance. Fourth revised and extended edition, De Gruyter (2016)
+* [10] Dellacherie, C.: Quelques commentaires sur les prolongements de capacites. Seminaire Probabilites V, Strasbourg, Lecture Notes in Math., vol. **191**, Springer-Verlag, Berlin and New York (1970)
+* [11] Gal, S.G.: Uniform and pointwise quantitative approximation by Kantorovich-Choquet type integral operators with respect to monotone and submodular set functions. Mediterr. J. Math. **14**, 205-216 (2017)
+* [12] Gal, S.G.: Quantitative approximation by Stancu-Durrmeyer-Choquet-Sipos operators, Math. Slovaca **69**(3), 625-638 (2019)
+* [13] Gal, S.G., Niculescu, C.P.: Kantorovich's mass transport problem for capacities. Proc. Rom. Acad. Ser. A Math. Phys. Tech. Sci. Inf. Sci. **20**(6), (2019)
+* [14] Gal, S.G., Opris, B.D.: Uniform and pointwise convergence of Bernstein-Durrmeyer operators with respect to monotone and submodular set functions. J. Math. Anal. Appl. **424**, 1374-1379 (2015)
+* [15] Gal, S.G., Trifa, S.: Quantitative estimates in uniform and pointwise approximation by Bernstein-Durrmeyer-Choquet operators. Carpath. J. Math. **33**, 49-58 (2017)
+* [16] Grabisch, M.: Set Functions, Games and Capacities in Decision Making. Springer, Berlin (2016)
+* [17] Grossman, M. V.: Note on a Generalized Bohman-Korovkin Theorem. J. Math. Anal. Appl. **45**, 43-46 (1974)
+* [18] Korovkin, P.P.: On convergence of linear positive operators in the space of continuous functions (Russian). Doklady Akad. Nauk. SSSR (NS) **90**, 961-964 (1953).
+* [19] Korovkin, P.P.: Linear Operators and Approximation Theory. Hindustan Publ. Corp., Delhi (1960).
+* [20] Niculescu, C.P.: Absolute continuity in Banach space theory. Rev. Roum. Math. Pures Appl. **24**, 413-423 (1979).
+* [21] Niculescu, C.P.: An overview of absolute continuity and its applications. Internat. Ser. Numer. Math., **157**, pp. 201-214, Birkhauser, Basel (2009).
+
+* [22] Schaefer, H.H.: Banach Lattices and Positive Operators. Springer-Verlag, Berlin (1974)
+* [23] Schempp, W.: A note on Korovkin test families. Arch. Math. (Basel) **23**, 521-524 (1972)
+* [24] Shafer, G.: Allocations of probability. Ann. of Probability **7**(5), 827-839 (1979)
+* [25] Volkov, V.I.: On the convergence of sequences of linear positive operators in the space of continuous functions of two variables (Russian). Dokl. Akad. Nauk. SSSR (N.S.) **115**, 17-19 (1957)
+* [26] Wang, Z., Klir, G.J.: Generalized Measure Theory. Springer-Verlag, New York (2009)
+* [27] Wang, Z., Yan, J.-A.: A selective overview of applications of Choquet integrals. Advanced Lectures in Mathematics, pp. 484-515, Higher Educational Press and International Press (2007)
+* [28] Zhou, L.: Integral representation of continuous comonotonically additive functionals. Trans. Amer. Math. Soc. **350**, 1811-1822 (1998)

2003.00003v1.mmd

@@ -0,0 +1,262 @@
+# Toward Active and Passive Confidentiality Attacks On Cryptocurrency Off-Chain Networks
+
+Utz Nisslmueller\({}^{1}\)
+
+Stefan Schmid\({}^{1}\)
+
+\({}^{1}\)Faculty of Computer Science, University of Vienna, Vienna, Austria
+
+\({}^{2}\)Blockstream, Zurich, Switzerland
+
+Klaus-Tycho Foerster\({}^{1}\)
+
+Christian Decker\({}^{2}\)
+
+\({}^{1}\)Faculty of Computer Science, University of Vienna, Vienna, Austria
+
+\({}^{2}\)Blockstream, Zurich, Switzerland
+
+###### Abstract
+
+Cryptocurrency off-chain networks such as Lightning (e.g., Bitcoin) or Raiden (e.g., Ethereum) aim to increase the scalability of traditional on-chain transactions. To support nodes in learning about possible paths to route their transactions, these networks need to provide gossip and probing mechanisms. This paper explores whether these mechanisms may be exploited to infer sensitive information about the flow of transactions, and eventually harm privacy. In particular, we identify two threats, related to an active and a passive adversary. The first is a _probing attack:_ here the adversary aims to detect the maximum amount which is transferable in a given direction over a target channel by actively probing it and differentiating the response messages it receives. The second is a _timing attack:_ the adversary discovers how close the destination of a routed payment actually is, by acting as a passive man-in-the middle and analyzing the time deltas between sent messages and their corresponding responses. We then analyze the limitations of these attacks and propose remediations for scenarios in which they are able to produce accurate results.
+
+## 1 Introduction
+
+Blockchains, the technology underlying cryptocurrencies such as Bitcoin or Ethereum, herald an era in which mistrusting entities can cooperate in the absence of a trusted third party. However, current blockchain technology faces a scalability challenge, supporting merely tens of transactions per second, compared to custodian payment systems which easily support thousands of transactions per second. This is the result of the underlying global consensus algorithms, which tread on the side of correctness rather than performance.
+
+Off-chain networks [9], a.k.a. payment channel networks (PCNs) or second-layer blockchain networks, have emerged as a promising solution to mitigate theblockchain scalability problem: by allowing participants to make payments directly through a network of _peer-to-peer_ payment channels, the overhead of global consensus protocols and committing transactions on-chain can be avoided. Off-chain networks such as Bitcoin Lightning [18], Ethereum Raiden [23], and XRP Ripple [8], to just name a few, promise to primarily reduce load on the underlying blockchain, as well as drastically increasing transaction throughput, and thus, being able to settle transactions in the matter of (sub-) seconds rather than in minutes or in hours - along with substantially reducing transaction fees, since now only one counterparty is responsible for validating a payment initially, rather than the whole network.
+
+In all of these networks, each node typically represents a user and each weighted edge represents funds escrowed on a blockchain; these funds can be transacted only between the endpoints of the edge. Many payment channel networks use source routing, in which the source of a payment specifies the complete route for the payment. If the global view of all nodes is accurate, source routing is highly effective because it finds all paths between pairs of nodes. Naturally, nodes are likely to prefer paths with lower per-hop fees, and are only interested in paths which support their transaction, i.e. which have a sufficient channel capacity.
+
+However, the fact that nodes need to be able to find routes also requires mechanisms for nodes to learn about the payment channel network's state. The two typical mechanisms which enable nodes to find and create such paths are _gossip_ and _probing_. The gossip protocol defines messages which are to be broadcast in order for participants to be able to discover new nodes and channels and keep track of currently known nodes and channels [15]. _Probing_ is the mechanism which is used to construct an actual payment route based on a local network view delivered by gossip, and ultimately perform the payment. In the context of SS4, we are going to exploit probing to discover whether a payment has occurred over a target channel. The gossip store is queried for viable routes to the destination, based on the desired route properties [25]. Because the gossip store contains global channel information, it is possible to query payment routes originating from any node on the network. Due to privacy concerns, gossip messages only include the _total_ balance for any given channel rather than the balance each node is holding.
+
+This paper explores the question whether the inherent need for nodes to discover routes in general, and the gossip and probing mechanisms in particular, can be exploited to infer sensitive information about the off-chain network and its transactions.
+
+### Our Contributions
+
+This paper identifies two novel threats for the confidentiality of off-chain networks. In particular, we consider the Lightning Network as a case study and present two attacks, an active one and a passive one. The active one is a _probing attack_ in which the adversary wants to determine the maximum amount which can be transferred over a target channel it is directly or indirectly connected to, by active probing. The passive one is a _timing attack_ in which the adversary discovers how close the destination of a routed payment actually is, by acting as a man-in-the middle and listening for / analyzing certain well-defined messages. We then analyze these attacks, identify limitations and also propose remediations for scenarios in which they are able to produce accurate results.
+
+### Organization
+
+Our paper is organized as follows. We introduce some preliminaries in SS2, and then first describe the probing attack in SS4 followed by the timing attack in SS5. We review related work in SS3 and conclude in SS6.
+
+## 2 Preliminaries
+
+While our contribution is applicable to the concept of off-chain networks in general, to be concrete, we will consider the Bitcoin Lightning Network (LN) as a case study in this paper. In the following, we will provide some specific preliminaries which are necessary to understand the remainder of this paper.
+
+The messages which are passed from one Lightning node to another are specified in the Basics of Lightning Technology (BOLTs) [17]. Each message is divided into a subcategory, called a layer. This provides superior separation of concerns, as each layer has a specific task and, similarly to the layers found in the Internet Protocol Suite, is agnostic to the other layers.
+
+For example in Lightning, the channel_announce and channel_update messages are especially crucial for correct payment routing by other nodes on the network. channel_announce signals the creation of a new channel between two LN nodes and is broadcast exactly once.
+
+channel_update is propagated at least once by each endpoint, since even initially each of them may have a different fee schedule and thus, routing capacity may differ depending on the direction the payment is taking (i.e., when \(c\) is the newly created channel between A and B, whether \(c\) is used in direction AB or BA). Once a viable route has been determined, the sending node needs to construct a message (a transaction "request") which needs to be sent to the first hop along the route. Each payment request is accompanied by an onion routing packet containing route information. Upon receiving a payment request each node strips one layer of encryption, extracting its routing information, and ultimately preparing the onion routing packet for the next node in the route. For the sake of simplicity, cryptographic aspects are going to be omitted for the rest of this chapter. We refer to [14] and [16] for specifics.
+
+Two BOLT Layer 2 messages are essential in order to to establish a payment chain:
+
+* update_add_htlc: This message signals to the receiver, that the sender would like to establish a new HTLC (Hash Time Locked Contract), containing a certain amount of millisatoshis, over a given channel. The message also contains an onion_routing_packet field, which contains information to be forwarded to the next hop along the route. In Figure 1, the sender initially sets up an HTLC with Hop 1. The onion_routing_field contains another update_add_htlc (set up between Hop 1 and Hop 2), which in turn contains the ultimate update_add_htlc (set up between Hop 2 and Destination) in the onion_routing_field.
+* update_fulfill_htlc: Once the payment message has reached the destination node, it needs to release the payment hash preimage in order to claim the funds which have been locked in the HTLCs along the route by the forwarded update_add_htlc messages. For further information on why this is necessary and how HTLCs ensure trustless payment chains, see [4]. To achieve this, the preimage is passed along the route backwards, thereby resolving the HTLCs and committing the transfer of funds (see Steps 4, 5, 6 in Figure 1).
+
+The gossip messages mentioned earlier are sent to every adjacent node and eventually propagate through the entire network.
+
+update_add_htlc and update_fulfill_htlc however, are only sent/forwarded to the node on the other end of the HTLC.
+
+In order to test the attacks proposed in SS4 and SS5, we have set up a testing network consisting of four c-lightning [2] nodes, with two local network computers running two local nodes each (Figure 2). Nodes 1 and 2 are connected via a local network link and can form hops for payment routes between Nodes
+
+Figure 1: An exemplary transaction from source to destination, involving two intermediate nodes.
+
+3 and 4. In order to interact with the nodes, we have made use of c-lightning's RPC interface and built our software tool set in Python [19]. The tests and their corresponding results in 5 have also been verified with LND [3], another BOLT-conform Lightning Network implementation, written in Go.
+
+## 3 Related Work
+
+Off-chain networks in general and the Lightning network in particular have recently received much attention, and we refer the reader to the excellent survey by Gudgeon et al. [9]. The Lightning Network as an second-layer network alternative to pure on-chain transactions was first proposed by [22], with the technical specifications laid out in [18]. Despite being theoretically currency-agnostic, current implementations such as c-lightning [2] and LND [3] support BTC exclusively. A popular alternative for ERC-20 based tokens is the Raiden Network [23].
+
+Several papers have already analyzed security and privacy concerns in off-chain networks. Rohrer et al. [24] focus on channel-based attacks and proposes methods to exhaust a victim's channels via malicious routing (up to potentially total isolation from the victim's neighbors) and to deny service to a victim via malicious HTLC construction. Tochner et al. [27] propose a denial of service attack by creating low-fee channels to other nodes, which are then naturally used
+
+Figure 2: Local Testing Setup
+
+to route payments for fee-minimizing network participants and then dropping the payment packets, therefore forcing the sender to await the expiration of the already set-up HTLCs.
+
+[10] provides a closer look into the privacy-performance trade-off inherent in LN routing. The authors also propose an attack to discover channel balances within the network. Wang et al. [28] examine the LN routing process in more detail and proposes a split routing approach, dividing payments into large size and small size transactions. The authors show that by routing large payments dynamically to avoid superfluous fees and by routing small payments via a lookup mechanism to reduce excessive probing, the overall success rate can be maintained while significantly reducing performance overhead. Beres et al. [6] make a a case for most LN transactions not being truly private, since their analysis has found that most payments occur via single-hop paths. As a remediation, the authors propose partial route obfuscation/extension by adding multiple low-fee hops. Currently still work in progress, [5] is very close to [4] in its approach and already provides some insights into second-layer payments, invoices and payment channels in general. The Lightning Network uses the Sphinx protocol to implement onion routing, as specified in [14]. The version used in current Lightning versions is based on [7] and [11], the latter of which also provides performance comparisons between competing protocols.
+
+## 4 Probing Attack
+
+### Design
+
+The Lightning Network uses an invoice system to handle payments. A LN invoice consists of a destination node ID, a label, a creation timestamp, an expiry timestamp, a CLTV (Check Lock Time Verify) expiry timestamp and a payment hash. Paying an invoice with a randomized payment hash is possible (since the routing nodes are yet oblivious to the actual hash) and will route the payment successfully to its' destination, which forms the basis of this attack. Optionally it can contain an amount (leaving this field empty would be equal in principle to a blank cheque), a verbal description, a BTC fallback address in case the payment is unsuccessful, and a payment route suggestion. This invoice is then encoded, signed by the payee, and finally sent to the payer.
+
+Having received a valid invoice (e.g. through their browser or directly via e-mail), the payer can now either use the route suggestion within the invoice or query the network themselves, and then send the payment to the payee along the route which has been determined. In this section, we will use the c-lightning RPC interface via Python exclusively - the functions involved are getroute() [25] and sendpay() [26], which takes two arguments: the return object from a getroute() call for a given route, a given amount and a given riskfactor, as well as the payment hash. Using sendpay() on its own (meaning,with a random payment hash instead of data from a corresponding invoice) will naturally result in one of two following error codes:
+
+* **204 (failure along route):** This error indicates that one of the hops was unable to forward the payment to the next hop. This can be either due to insufficient funds or a non-existent connection between two adjacent hops along the specified route. If we have ensured that all nodes are connected as depicted in Figure 2, we can safely assume the former. One sequence of events leading up to this error can be seen in Figure 3. 
+* **16399 (permanent failure at destination):** Given the absence of a 204 error, the attempted payment has reached the last hop. As we are using a random payment hash, realistically the destination node will throw an error, signalling that no matching preimage has been found to produce the payment hash. The procedure to provoke a 16399 error code can be seen in Figure 4. 
+
+Figure 3: Causing a 204 error by trying to send a payment to Node 4, which Node 3 is unable to perform.
+
+The goal of this attack is to trace payment flow over a channel, which the attacker node is directly or indirectly connected to. The attacker node will therefore initially attempt to determine whether a payment has occurred over the observed channel between the penultimate and final node along the route. To this end, the attacker will send out periodic probes to the final node (the "victim"), containing the amount which has been determined by the initial probe. If channel weights remain unaltered, each of these probes should return a 16399 error code. If a payment does occur however, the penultimate node will find itself unable to forward the payment on the outgoing channel to our target, yielding a 204 error response. Upon receiving this message, we can then restart the process of our initial probe and ultimately arrive at the exact amount of millisatoshis (msat), which have been transferred.
+
+### Lab Implementation
+
+Recalling Figure 2, we have chosen Node 3 as our attacker node and Node 4 as our target node - hence, the initial goal of Node 3 is to determine the max
+
+Figure 4: Causing a 16399 error by trying to send a payment to Node 4, who can’t produce a matching preimage and thus fails the payment.
+
+imum payment flow between Nodes 2 and 4. To conduct our tests, each of the channels has been set up with a balance of 200,000,000 msat, with each node holding a stake of 100,000,000 msat in each of its channels. Node 3 will hold a slightly higher balance in order to accommodate probing fees. We can use the total channel balance, as received via gossip, as an upper ceiling for this value (200,000,000 msat in this case). We can then send payments from Node 3 to Node 4 with random payment hashes - resulting in either error code 16399 or error code 204 (SS4.1). To this end, we perform a binary search on the available funds which we can transfer, searching for the highest value yielding a 16399 error instead of a 204 error. The algorithms used for both initial probing and deriving the actual channel balance from Node 2 to Node 4 are depicted in Algorithms 1 and 2.
+
+```
+Result: Either error code 204 or 16399 payment_hash = random.hex(); node_id = node ID of final node on victim channel; msat = value to probe for; route = getroute(node_id, msat); sendpay(route, payment_hash);
+```
+
+**Algorithm 1**Probing a channel for a given amount of msat
+
+```
+Result:amount_msat - initial channel balance min_msat = 0; max_msat = channel.balance; amount_msat = channel.balance / 2; whileTruedo ifprobe(amount_msat) == 16399then min_msat = amount_msat; else ifamount_msat) == 204then max_msat = amount_msat; else return "No suitable route found.";  end if ifmax_msat - min_msat! 1000then return amount_msat; else // continue to minimise maximum error end if amount_msat = (min_msat + max_msat) / 2 end
+```
+
+**Algorithm 2**Finding the initial maximum channel balance
+
+We thus arrive at the approximate maximum amount, which Node 2 can transfer to Node 4. The next step is to continuously probe for this amount of msat in regular intervals. The expected response is a 16399 error code, with a 204 error code implying that the amount we are trying to send is higher than the available amount which Node 2 can transfer to Node 4 (or that it has disconnected from Node 4). Upon receiving a 204 response, we start looking for the maximum payable amount to Node 4 once more. Subtracting the new amount from the old amount, we arrive at the size of the transaction which has occurred between Nodes 3 and 4.
+
+After 17 probes by Node 3, Algorithm 2 has yielded an initial balance of 99,999,237 msat, which is in line with the channel balance we have allocated between Nodes 2 and 4. The next step is to monitor the channel for potential weight changes (Algorithm 3).
+
+```
+Result: New maximum flow from penultimate to final node init_max = initial channel balance; new_max = init_max; t = time to wait between checks; whileTruedo  sleep(t); ifprobe(init_max) == 204then  // channel balance has decreased  return find_init_max(); // potentially calculate delta else if(init_max + 1000) == 16399then  // channel balance has increased  return find_init_max(); // potentially calculate delta else  return error;  end if  end if  end while
+```
+
+**Algorithm 3**Finding the initial maximum channel balance
+
+To verify this, we have transferred 50,000,000 msat from Node 2 to Node 4, with our program detecting this soon after (we have set t to 5 seconds in order to avoid excessive probing) and returning an updated balance of 49,998,237 msat. We then transferred another 30,000,000 msat from Node 1 to Node 4, with our program again picking up the change and reporting the new channel balance at 19,997,389 msat.
+
+Figure 5 shows the trade-off between probing run time and the error in the channel balance estimate we observed for test runs on our lab setup. As we
+
+wanted to avoid overly excessive probing while conducting our tests, we were generally satisfied with any answer which is less than 1000 msat (the actual minimum BTC denomination) lower than the actual channel balance. Another possible approach could be keeping the number of probes sent out to the target constant, hence providing a more uniform level of balance error and probing duration.
+
+### BTC Testnet Evaluation
+
+For analysis on the feasibility of our attack over the BTC Testnet, we connected Nodes 1, 2 and 3 from Figure 2 to the "ion.radar.tech" Testnet Lightning node. We chose this host in particular, since their website allowed us to alter the channel weights by generating payable invoices with parameters of our choosing. The exact connections along with the corresponding channel weights can be seen in Figure 6. Our goal was to verify the results we obtained in 4.2 and see whether probing duration (see Figure 5) was affected by the public Testnet hop in place of the local hop(s) used in 4.2. Running an initial series of probes from Node 3 to Node 1, we arrived at a channel balance of 149,926,757 msat between the radar.ion.tech node and Node 1 (99.95% accuracy). We attribute this comparatively high error in regard to our tests in 4.2 due to the Testnet nodes' differing fee structure, which is necessarily taken into account when constructing the payment route. Then, we sent a payment containing 50,000,000 msat from Node 2 to Node 1 - predictably, Node 3 returned the updated maximum payment flow on the observed channel correctly with 99,902,343 msat (99.9 % accuracy).
+
+Figure 5: Visualizing the trade-off between probing accuracy and duration.
+
+After verifying the correct operation of our program for 16399 error codes, we were been on discovering whether 204 error code scenarios would be dealt with correctly as well. In order to test this, we transferred back any amounts which have been redistributed as part of our initial test, increased the channel balance between Node 1 and radar.ion.tech by a factor of 10 and modified the setup from Figure 6 slightly by placing an intermediary hop between radar.ion.tech and Nodes 2 and 3. The updated infrastructure can be seen in Figure 7.
+
+It became apparent however, that we would need to rethink the weights we allotted to the respective nodes, as we were initially unaware of the true channel weights between the radar.ion.tech and "lnd.vanilla.co.za" nodes. Naturally, we were inclined to simply run the find_init_max() function (Algorithm 2) from Node 3 on the ion.radar.tech node. However, we found that the two
+
+Figure 6: Setup and balance allocations of our first testnet evaluation (balances given in satoshis).
+
+Figure 7: Initial setup and balance allocations of our second Testnet evaluation (balances given in satoshis).
+
+nodes were connected by 6 channels rather than one. To circumvent this route ambiguity, we queried a route for 1,000, 1,000,000 an 1,000,000,000 msat using default parameters, hoping all of them would return the same route, thus allowing us to treat the resulting channel as the only one connecting these two nodes. Unfortunately though, we received varying responses for all of these amounts, introducing a large uncertainty in any subsequent measurements. We then tried to run our tests on these channels, with all of them reporting failure in establishing a route to the target. We are not sure why even the initial probes failed and only further analysis and testing of our program will unveil the error in our approach. We decided to conclude our Testnet evaluation at this point, since despite extensive refactoring, we were not able to produce meaningful results for this constellation of nodes and channels, leaving route ambiguity and handling of multiple channels to be explored by further research in this area.
+
+### Results, Implications, and further Considerations
+
+In SS4.2 we have demonstrated that it is in fact possible to trace channel payments if the network is structured in a certain way. In theory, this method should hold true for any node which is reachable from the attacking node and has only one channel whose balance is lower or or equal to the second lowest balance on the route from the attacking node. We have partially verified this supposition in 4.3 while maintaining a high accuracy in our successful measurements. This is particularly a threat to end users, since most of them connect to a single well-connected node over a single channel, in order to interact with the rest of the network [1]. Nonetheless, there are several caveats to this method, the most significant of which are:
+
+* **Excluding the possibility of payment forwarding:** The attack laid out in this chapter does not take into account the fact that nodes can be used to forward payments. Hearkening back to Figure 2, if we were to select the channel between Nodes 1 and 2 as our target, transactions between Nodes 1 and 4 would appear as if they were transactions to Node 3. One opportunity of accounting for this would be to monitor _every_ channel to and from Node 3 for changes in directed channel balance, which would create problems on its own (see below).
+* **Surge of unresolved HTLCs while probing:** Recalling steps 5-7 in Figure 4, each probe sets up a chain of irredeemable HTLCs (since a matching preimage would have to be brute-forced). Eventually, running multiple probes over the same channels will escrow its funds in these HTLCs, effectively DOSing the probe route and forcing the nodes to wait until the HTLCs time out before being able to forward other payments. This is an issue we encountered over and over during 4.2 and 4.3, often giving us one shot at probing before having to wait multiple hours for the HTLCs to expire. This is also why we chose the channels leading up to our final target to have a much higher balance, so that we would have enough balance left after initial probing to monitor the channel for a reasonable period of time.
+* **Insufficient sensitivity for high-frequency transactions:** Looking back at Algorithm 3, we have defined the parameter t as the time, for which to wait during probes for monitoring the channel balance, one the initial maximum value has been discovered. If more than one transaction would occur during this timeframe, it would still only show up as a singular payment with our tool. In the worst-case scenario, two transactions covering the same amount could take place in opposite directions, not changing the weighted balance at all and thus eluding our detection mechanisms.
+* **Omission of private channels:** Upon creating a channel, the node can declare the channel as private, and thus prevent it from being broadcast via gossip. The channel is fully functional for both nodes which are connected by it, but no foreign payments can be routed through it. Looking ahead to increasing adoption of the Lightning Network, this provides an intriguing opportunity for nodes, which do not wish to participate in routing (e.g. mobile wallets) or nodes with limited uptime (personal computers). Routing would only occur between aggregating nodes (such as payment providers), with most of the channels (and therefore nodes) on the network remaining invisible to malicious participants as the gossip protocol would only propagate public channels. This further exacerbates our ability to detect forwarded payments (see above) as opposed to actual payments, since private channels can't be monitored by design.
+* we would like to point the interested reader to [27] for suggestions on route hijacking and thus effectively bypassing the bottleneck along the route.
+
+During the tests we conducted in 4.3, we also encountered the hops between Node 1 and Node 3 being connected via multiple channels. As confirmed by our observations, it is entirely possible to receive varying routes for differing amount_msat, riskfactor, cltv and fuzzpercent [25] combinations. Our tool failed to produce accurate results in this scenario, as it was designed assuming singular channels between pairs of nodes. It is however perfectly reasonable to have multiple channels between two nodes, as channel balances are final and can't be increased after creation. We expect this to be the predominant form of retrospectively increasing potential payment flow between nodes and further research on how to deal with this complication would be highly appreciated.
+
+All in all, the probing attack we laid out in this chapter can be seen more as a proof of concept rather than a realistic attack vector, due to the limitations discussed in 4.4. We are confident that certain aspects such as the exact algorithm and route construction could be refined to provide more reliable results. However other aspects such as the binding of channel funds in irredeemable HTLs and the incomplete network view due to private channels provide a much more consistent barrier to uncovering payment flows in real-world scenarios.
+
+## 5 Timing Attack
+
+### Design
+
+The Lightning Network is often referred to as a payment channel network (PCN). Performing payments over multiple hops is possible due to the use of HTLC's [22], a special bitcoin transaction whose unlocking conditions effectively rid the Lightning Network and its users of all trust requirements. An exemplary chain of HTLs along with their shortened unlocking conditions is shown in Figure 8. Note that any node can only retrieve the funds locked in the HTLs if they share R, and that each HTLC starting from Node 4 is valid for 2 hours longer than the previous HTLC to provide some room for error/downtime.
+
+Due to the Onion Routing properties of the Lightning Network, it is cryptographically infeasible to try and determine where along the route a forwarding node is located, since each node can only decrypt the layer which was intended for it to decrypt. Attempts to analyze the remaining length of the routing packet have been thwarted at the protocol level by implementing a fixed packet size with zero padding at the final layer [14].
+
+The only opportunity left to analyze the encrypted traffic between the nodes is to extract time-related information from the messages. One possibility would be to analyze the cltv_expiry_delta field (analogous to "hours passed" in Figure 8, measured in mined blocks since the establishment of the HTLC): By looking at the delay of both the incoming and the outgoing HTLC, a node could infer how many hops are left until the payment destination. However, this possibility has been accounted for by the adding "shadow routes" to the actual payment path, with each node fuzzing path information by adding a random offset to the cltv_expiry_delta value, hence effectively preventing nodes from guessing their position along the payment route [15].
+
+The method we propose, is to time messages at the network level, rather than at the protocol level (e.g. through cltv_expiry_delta). Recalling Figure 8, Node 2 can listen for response messages from Node 4, since there is currently no mechanism in place to add delay to update_fulfill_htlc responses (in fact,[13] states that "_a node SHOULD remove an HTLC as soon as it can_"). Based on response latency, Node 2 could infer its position along the payment route to a certain extent, as examined in SS5.2.
+
+### Lab Implementation
+
+Initial analysis has shown that analyzing packets directly (e.g. via Wireshark) is of little avail, since LN messages are end-to-end encrypted - meaning that even if we know the target nodes' IP address and port number, we can not detect the exact nature of the messages exchanged. We hence chose to redirect the output of the listening c-lightning node to a log file, which we then analyze with a Python script. As in SS4, the source code can be found at [19].
+
+Looking at the log file, we are particularly interested in the two messages discussed in SS2: update_add_htlc and update_fulfill_htlc. The node output includes these events, complete with timestamps and the corresponding node ID with which the HTLC is negotiated. By repeatedly sending money back and forth between Nodes 1 and 3 in our test setup (Figure 2), we arrive at a local (and therefore minimum) latency of 182ms on average. The latency distribution
+
+Figure 8: Paying a LN invoice over multiple hops. Messages 2-4 are update_add_htlc messages, messages 7-9 are update_fulfill_htlc messages.
+
+for small (1,000 msat) payments can be seen in Figure 9. We have found that latencies remain largely unaffected by transaction size - increasing payment size by a factor of 100,000 actually slightly reduced average settlement time and standard deviation (Figure 10).
+
+Next, we examined whether an increase in hop distance would yield predictable results. To this end, we first timed payments over 1 network hop from Node 2 to Node 1 (Figure 11). Then, we timed payments over the same amount over 1 network and 1 local hop from Node 2 to Node 3 (Figure 12). Based on these results, we derive that timing messages on a local network with little to no interfering traffic scales predictably over several hops, with 1 network hop roughly corresponding to 1.284 local hops in terms of latency.
+
+### BTC Testnet Evaluation
+
+Building on the results obtained in 5.2, we were keen to discover whether the they would carry over into real-world evaluations. To this end, we connected Node 1 and Node 3 from Figure 2 to the "endurance" Lightning Testnet node. Located in Dublin, Ireland and being connected to over 500 other Lightning Testnet nodes [1], we concluded that this node would provide a good entry point to test network latency from our location in Vienna, Austria, with the possibility to construct longer and more complicated routes over it as we saw fit. In order to constitute an initial RTT value, we established an HTTP connection to Lightning's default port 9735 [12], since the target host appeared to drop our ICMP ping requests. Alternating our requests between Systems A
+
+Figure 9: Latency times for local payments containing 1,000 msat (\(\mu\) = 0.1852, \(\sigma\) = 0.0974, \(n\) = 25)
+
+and B (Figure 2) in an attempt to prevent cached responses, we have found that HTTP response times were fairly constant from this node, with an average response time of 0.067s (\(\sigma\) = 0.0206).
+
+Next, we were interested whether payments over the public hop were sub
+
+Figure 11: Latency times for payments containing 100,000,000 msat over 1 network hop (\(\mu\) = 0.234, \(\sigma\) = 0.025, \(n\) = 25)
+
+Figure 10: Latency times for local payments containing 100,000,000 msat (\(\mu\) = 0.1798, \(\sigma\) = 0.0385, \(n\) = 25)
+
+ject to an equally uniform latency as in 5.2. Thus, we created 25 invoices over 1,000,000 msat each (having found in 5.2 that response latency is independent of payment size) at Node 3 and sent the payment from Node 1. As seen in Figure 13, the fulfill message response times were remarkably consistent, however latency did not scale to our expectations. Based on Figure 12, we expected to be overall latency to be in the ballpark of 0.5-0.7 seconds (2x local network RTT + HTTP request RTT), however actual latency was twice that value. Results from the aranguren.org Testnet node, located in Melbourne, Australia, proved equally consistent with an average ping time of 0.314s (\(\sigma\) = 0.035) and an average HTLC fulfillment latency of 1.68s (\(\sigma\) = 0.0972)
+
+Finally, we were curious about HTLC fulfillment delays over 2 public hops. To this end, we closed the channel between Node 3 and endurance and opened a new channel to the "aranguren.org" Testnet node, which in turn has a channel with endurance and thus re-establishes the chain of channels from Node 1 to Node 3. Timing results for this route can be seen in Figure 14. This marked the end of our timing tests, since we were not able to establish an acyclic payment route over 3 or more publicly available LN nodes. This coincides with the observation that neither the attempted nor the actual payments we performed during the course of 4 and 5 were routed over more than two public hops.
+
+### Results, Implications and further Considerations
+
+Considering the findings in SS5.2, we can see that timing produces fairly reliable and uniformly distributed results over a local network with little outside inter
+
+Figure 12: Latency times for payments containing 100,000,000 msat over 1 network hop and 1 local hop (\(\mu\) = 0.414, \(\sigma\) = 0.05, \(n\) = 25)
+
+ference. Yet, due to the nature of LN routing, it is not possible to determine the distance or path to the initial payment source. To our surprise however, RTT remained equally consistent over 1-2 internet hops. Data acquired during monitoring of the local (mostly idle) network suggests that the timing node won't be able to distinguish traffic originating from a local node from the traffic
+
+Figure 14: Latency times for payments between Node 1 and Node 3 over the endurance and aranguren.org Lightning nodes (\(\mu\) = 2.3349, \(\sigma\) = 0.0475, \(n\) = 25)
+
+Figure 13: Latency times for payments between Node 1 and Node 3 over the endurance Lightning node (\(\mu\) = 1.0179, \(\sigma\) = 0.0542, \(n\) = 25)
+
+in SS5.2 without further information due to low latency deltas ranging from 2ms to 5ms.
+
+While performing timing measurements for payments across the BTC Testnet network, we have found that HTLC settlement takes long enough over even 1 hop to make traffic RTT volatility negligible. Over 1 hop, we conclude that HTLC settlement for our Vienna-based node should be in the ballpark of 0.86 - 1.97 seconds with 2-hop latency amounting to roughly 1.99 - 2.68 seconds, depending on the geographical location and assuming a normal distribution for the measured latency deltas. Further research could include a further statistical examination of the ability to differentiate distances for HTLC deltas at the sub-2-second threshold. We suggest that overall network bandwidth does not affect the acquired results significantly, since after performing all payments in 4, Node 1 has sent 64 KB and Node 3 has received 55 KB - only a fraction of which were outgoing/incoming HTLCs (alongside gossip, pings, etc.).
+
+Our results open many new avenues for further timing-based research on the Lightning Network. The next step for us would be to develop a tool to predict the distance to the final destination of an HTLC which is passing through the listening node, based on the measurements laid out in 5.3. It would be interesting to see whether there is a possibility to force payment-unrelated response messages, e.g. by forging ping messages [12] in order to estimate (possibly network-wide) RTTs, correlate HTLC settlement latencies against them and finally arrive at a set of nodes which must have been the ultimate recipient of the forwarded payment. Furthermore, experiments could be conducted on the feasibility of adding a random time offset to HTLC fulfillments, and the trade-offs involved therein.
+
+## 6 Conclusion
+
+This paper has shown that off-chain routing and payment settlement mechanisms may be exploited to infer confidential information about the network state. In particular, considering the Lightning Network with Bitcoin as the underlying blockchain as a case study, we set up a local infrastructure and proposed two ways in which two current state-of-the-art implementations, c-lightning and LND, can be exploited to gain knowledge about distant channel balances and transactions to unconnected nodes: By deliberately failing payment attempts, we were able to deduce the exact amount of (milli-)satoshis on a channel located two hops away on our local lab infrastructure. Using this technique repeatedly, we were able to determine whether a transaction occurred between one node and another over the monitored channel. To a certain extent, we were able to reproduce these results in the public Bitcoin Testnet chain. We also identified this attacks' limitations and proposed some workarounds to these obstacles.
+
+By timing the messages related to HTLC construction and termination, we were able to infer the remaining distance of a forwarded packet accurately in our test lab. These results transferred well into our Testnet evaluation, while being free of the partially restrictive limitations which we discovered during our examination of the probing attack. We concluded that RTT volatility of the HTLC message cycle was low enough for public Testnet hops which were within geographical vicinity to our node in Vienna, Austria, as well as for hops which were located in East Asia, to establish latency approximate latency boundaries for the number of remaining hops along the payment route of a forwarded transaction.
+
+Our work raises several interesting research questions. In particular, it remains to fine-tune our attacks, to improve the flexibility of our software tools and to finally conduct more systematic experiments including more natural/interconnected network topologies, particularly on other off-chain networks. More generally, it will be interesting to explore further attacks on the confidentiality of off-chain networks exploiting the routing mechanism and investigate countermeasures. Furthermore, our work raises the question whether such vulnerabilities are an inherent price of efficient off-chain routing or if there exist rigorous solutions.
+
+**Bibliographical Note.** A preliminary version of this article appears at ICISSP 2020 [21]. The work herein is based on the thesis of Utz Nisslmueller [20].
+
+## References
+
+* Bitcoin Lightning Analysis Engine. [https://lml.com/](https://lml.com/), 2019. [Online; accessed 10-November-2019].
+* [2] c-lightning GitHub Repository. [https://github.com/ElementsProject/lightning](https://github.com/ElementsProject/lightning), 2019. [Online; accessed 26-December-2019].
+* [3] LND GitHub Repository. [https://github.com/lightningnetwork/lnd](https://github.com/lightningnetwork/lnd), 2020. [Online; accessed 18-January-2020].
+* [4]Antonopoulos, A. M. _Mastering Bitcoin: Unlocking Digital Crypto-Currencies_, 1st ed. O'Reilly Media, Inc., 2014.
+* [5]Antonopoulos, A. M., Osuntokun, O., and Pickhardt, R. Mastering the Lightning Network. [https://github.com/lnbook/lnbook](https://github.com/lnbook/lnbook), 2019. [Online; accessed 22-November-2019].
+* [6]Bees, F., Seres, I. A., and Benczur, A. A. A cryptoeconomic traffic analysis of bitcoins lightning network. _arXiv abs/1911.09432_ (2019).
+* [7]Danezis, G., and Goldberg, I. Sphinx: A compact and provably secure mix format. In _IEEE Symposium on Security and Privacy_ (2009), IEEE Computer Society, pp. 269-282.
+
+* [8]Fugger, R. Money as IOUs in social trust networks & a proposal for a decentralized currency network protocol. _Hypertext document. Available electronically at http://ripple. sourceforge. net 106_ (2004).
+* [9]Gudgeon, L., Moreno-Sanchez, P., Roos, S., McCorry, P., and Gervais, A. Sok: Off the chain transactions. _IACR Cryptology ePrint Archive 2019_ (2019), 360.
+* [10]Herrera-Joancomarti, J., Navarro-Arribas, G., Pedrosa, A. R., Perez-Sola, C., and Garcia-Alfaro, J. On the difficulty of hiding the balance of lightning network channels. In _AsiaCCS_ (2019), ACM, pp. 602-612.
+* [11]Kate, A., and Goldberg, I. Using sphinx to improve onion routing circuit construction. In _Financial Cryptography_ (2010), vol. 6052 of _Lecture Notes in Computer Science_, Springer, pp. 359-366.
+* [12]Lightning Network. BOLT 1: Base Protocol. [https://github.com/lightningnetwork/lightning-rfc/blob/master/01-messaging.md](https://github.com/lightningnetwork/lightning-rfc/blob/master/01-messaging.md), 2019. [Online; accessed 23-January-2020].
+* [13]Lightning Network. BOLT 2: Peer Protocol for Channel Management. [https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md](https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md), 2019. [Online; accessed 6-January-2020].
+* [14]Lightning Network. BOLT 4: Onion Routing Protocol. [https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md](https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md), 2019. [Online; accessed 3-January-2020].
+* [15]Lightning Network. BOLT 7: P2P Node and Channel Discovery. [https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md](https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md), 2019. [Online; accessed 4-December-2019].
+* [16]Lightning Network. BOLT 8: Encrypted and authenticated transport. [https://github.com/lightningnetwork/lightning-rfc/blob/master/08-transport.md](https://github.com/lightningnetwork/lightning-rfc/blob/master/08-transport.md), 2019. [Online; accessed 4-January-2020].
+* [17]Lightning Network. Lightning Network Specifications. [https://github.com/lightningnetwork/lightning-rfc/](https://github.com/lightningnetwork/lightning-rfc/), 2019. [Online; accessed 29-November-2019].
+* [18]Lightning Network. Lightning RFC: Lightning Network Specifications. [https://github.com/lightningnetwork/lightning-rfc](https://github.com/lightningnetwork/lightning-rfc), 2019. [Online; accessed 18-November-2019].
+* [19]Nisslmueller, U. Python code repository. [https://github.com/utzn42/icissp_2020_lightning](https://github.com/utzn42/icissp_2020_lightning), 2020. [Online; accessed 02-January-2020].
+
+* [20]Nisslmueller, U. Toward active and passive confidentiality attacks on cryptocurrency off-chain networks. Thesis at the University of Vienna, Austria, 2 2020.
+* [21]Nisslmueller, U., Foerster, K.-T., Schmid, S., and Decker, C. Toward active and passive confidentiality attacks on cryptocurrency off-chain networks. In _Proc. 6th International Conference on Information Systems Security and Privacy (ICISSP)_ (2020).
+* [22]Poon, J., and Dryja, T. The bitcoin lightning network: Scalable off-chain instant payments. [https://lightning.network/lightning-network-paper.pdf](https://lightning.network/lightning-network-paper.pdf), 2016. [Online; accessed 3-January-2020].
+* [23]Raiden Network. Raiden Network. [https://raiden.network/](https://raiden.network/), 2020. [Online; accessed 02-January-2020].
+* [24]Rohrer, E., Malliaris, J., and Tschorsch, F. Discharged payment channels: Quantifying the lightning network's resilience to topology-based attacks. In _EuroS&P Workshops_ (2019), IEEE, pp. 347-356.
+* [25]Russell, R. lightning-getroute Command for routing a payment (low-level). [https://lightning.readthedocs.io/lightning-getroute.7.html](https://lightning.readthedocs.io/lightning-getroute.7.html), 2019. [Online; accessed 6-December-2019].
+* [26]Russell, R. lightning-sendpay Low-level command for sending a payment via a route. [https://lightning.readthedocs.io/lightning-sendpay.7.html](https://lightning.readthedocs.io/lightning-sendpay.7.html), 2019. [Online; accessed 4-January-2020].
+* [27]Tochner, S., Schmid, S., and Zohar, A. Hijacking routes in payment channel networks: A predictability tradeoff. _arXiv abs/1909.06890_ (2019).
+* [28]Wang, P., Xu, H., Jin, X., and Wang, T. Flash: efficient dynamic routing for offchain networks. In _CoNEXT_ (2019), ACM, pp. 370-381.

2003.00004v1.mmd

@@ -0,0 +1,474 @@
+# Volterra-Choquet nonlinear operators
+
+Sorin G. Gal
+
+Department of Mathematics and Computer Science,
+
+University of Oradea,
+
+Universitatii Street No.1, 410087, Oradea, Romania
+
+E-mail: _galsorin23@gmail.com_
+
+and
+
+Academy of Romanian Scientists,
+
+Splaiul Independentei nr. 54
+
+050094 Bucharest, Romania
+
+###### Abstract
+
+In this paper we study to what extend some properties of the classical linear Volterra operators could be transferred to the nonlinear Volterra-Choquet operators, obtained by replacing the classical linear integral with respect to the Lebesgue measure, by the nonlinear Choquet integral with respect to a nonadditive set function. Compactness, Lipschitz and cyclicity properties are studied.
+
+**MSC(2010)**: 47H30, 28A12, 28A25.
+
+**Keywords and phrases**: Choquet integral, monotone, submodular and continuous from below set function, Choquet \(L^{p}\)-space, distorted Lebesgue measures, Volterra-Choquet nonlinear operator, compactness, Lipschitz properties, cyclicity.
+
+## 1 Introduction
+
+Inspired by the electrostatic capacity, G. Choquet has introduced in [5] (see also [6]) a concept of integral with respect to a non-additive set function which, in the case when the underlying set function is a \(\sigma\)-additive measure, coincides with the Lebesgue integral.
+
+Choquet integral is proved to be a powerful and useful tool in decision making under risk and uncertainty, finance, economics, insurance, pattern recognition, etc (see, e.g., [37] and [38] as well as the references therein).
+
+Many new interesting results were obtained as analogs in the framework of Choquet integral of certain known results for the Lebesgue integral. In this sense, we can mention here, for example, the contributions to function spaces theory in [4], to potential theory in [1], to approximation theory in [13]-[17] and to integral equations theory in [18], [19].
+
+Now, for \(1\leq p<+\infty\) denoting \(L^{p}[0,1]=\{f:[0,1]\to\mathbb{R};(L)\int_{0}^{1}|f(x)|^{p}dx<+\infty\}\), where the \((L)\) integral is that with respect to the Lebesgue measure, it is well-known that the classical Volterra linear operator introduced in 1896, is defined usually on \(L^{2}[0,1]\) by
+
+\[O(f)(x)=(L)\int_{0}^{x}f(t)dt,x\in[0,1]. \tag{1}\]
+
+Volterra operator has been studied and continues to be studied by many authors. The norm of Volterra operator is \(2/\pi\) (see the book [21], Problem 149). The Halmos' book also contains several nice results related with Volterra operator. The asymptotic behaviour of the norm \(\|V^{n}\|\) is described in [30]. An interesting fact about the Volterra operator is the determination of its invariant subspace lattice (see [7], Chapter 4 and [3], [9], [10], [23] and [32]). Compactness and cyclicity properties were studied in, e.g., [31], [20], [27], [28]. Very recent papers on various other aspects of the Volterra operator are, e.g., [29], [24], [2], [12], [26], to mention only a few. Note that there is also a huge literature dealing with the Volterra operator in complex setting, but this aspect is out of the discussions in the present paper.
+
+Let \(\mathcal{C}\) be a \(\sigma\)-algebra of subsets in \([0,1]\) and \(\mu:\mathcal{C}\to[0,+\infty]\) be a monotone set function, i.e. satisfying \(\mu(\emptyset)=0\) and \(\mu(A)\leq\mu(B)\) for all \(A,B\in\mathcal{C}\), with \(A\subset B\).
+
+The goal of the present paper is to study the possibilities of extension of the properties of classical Volterra linear operator, to the so-called Volterra-Choquet operator obtained by replacing the classical linear integral by the nonlinear Choquet integral, that is defined by
+
+\[V(f)(x)=(C)\int_{0}^{x}f(t)d\mu(t), \tag{2}\]
+
+where \(\mu\) is a set function not necessarily additive.
+
+The plan of the paper goes as follows. Section 2 contains preliminaries on the Choquet integral and Section 3 presents a few general preliminaries on compactness of nonlinear operators. In Section 4 we prove some compactness properties while in Section 5 we obtain some Lipschitz properties, for the Volterra-Choquet operators. Section 6 presents cyclicity properties for a Volterra-Choquet operator with respect to a particular distorted Lebesgue measure.
+
+## 2 Preliminaries on Choquet integral
+
+Some known concepts and results concerning the Choquet integral can be summarized by the following.
+
+**Definition 2.1.** Suppose \(\Omega\neq\emptyset\) and let \(\mathcal{C}\) be a \(\sigma\)-algebra of subsets in \(\Omega\).
+
+(i) (see, e.g., [37], p. 63) The set function \(\mu:\mathcal{C}\to[0,+\infty]\) is called a monotone set function (or capacity) if \(\mu(\emptyset)=0\) and \(\mu(A)\leq\mu(B)\) for all \(A,B\in\mathcal{C}\), with \(A\subset B\). Also, \(\mu\) is called submodular if
+
+\[\mu(A\bigcup B)+\mu(A\bigcap B)\leq\mu(A)+\mu(B),\mbox{ for all }A,B\in \mathcal{C}.\]\(\mu\) is called bounded if \(\mu(\Omega)<+\infty\) and normalized if \(\mu(\Omega)=1\).
+
+(ii) (see, e.g., [37], p. 233, or [5]) If \(\mu\) is a monotone set function on \(\mathcal{C}\) and if \(f:\Omega\to\mathbb{R}\) is \(\mathcal{C}\)-measurable (that is, for any Borel subset \(B\subset\mathbb{R}\) it follows \(f^{-1}(B)\in\mathcal{C}\)), then for any \(A\in\mathcal{C}\), the concept of Choquet integral is defined by
+
+\[(C)\int_{A}fd\mu=\int_{0}^{+\infty}\mu\left(F_{\beta}(f)\bigcap A\right)d\beta +\int_{-\infty}^{0}\left[\mu\left(F_{\beta}(f)\bigcap A\right)-\mu(A)\right]d\beta,\]
+
+where we used the notation \(F_{\beta}(f)=\{\omega\in\Omega;f(\omega)\geq\beta\}\). Notice that if \(f\geq 0\) on \(A\), then in the above formula we get \(\int_{-\infty}^{0}=0\).
+
+The function \(f\) will be called Choquet integrable on \(A\) if \((C)\int_{A}fd\mu\in\mathbb{R}\).
+
+(iii) (see, e.g., [37], p. 40) We say that the set function \(\mu:\mathcal{C}\to[0,+\infty]\) is continuous from below, if for any sequence \(A_{k}\in\mathcal{C}\), \(A_{k}\subset A_{k+1}\), for all \(k=1,2,...\), we have \(\lim_{k\to\infty}\mu(A_{k})=\mu(A)\), where \(A=\bigcup_{k=1}^{\infty}A_{k}\).
+
+Also, we say that \(\mu\) is continuous from above, if for any sequence \(A_{k}\in\mathcal{C}\), \(A_{k+1}\subset A_{k}\), for all \(k=1,2,...\), we have \(\lim_{k\to\infty}\mu(A_{k})=\mu(A)\), where \(A=\bigcap_{k=1}^{\infty}A_{k}\).
+
+In what follows, we list some known properties of the Choquet integral.
+
+**Remark 2.2.** If \(\mu:\mathcal{C}\to[0,+\infty]\) is a monotone set function, then the following properties hold :
+
+(i) For all \(a\geq 0\) we have \((C)\int_{A}afd\mu=a\cdot(C)\int_{A}fd\mu\) (if \(f\geq 0\) then see, e.g., [37], Theorem 11.2, (5), p. 228 and if \(f\) is of arbitrary sign, then see, e.g., [8], p. 64, Proposition 5.1, (ii)).
+
+(ii) In general (that is if \(\mu\) is only monotone), the Choquet integral is not linear, i.e. \((C)\int_{A}(f+g)d\mu\neq(C)\int_{A}fd\mu+(C)\int_{A}gd\mu\).
+
+In particular, for all \(c\in\mathbb{R}\) and \(f\) of arbitrary sign, we have (see, e.g., [37], pp. 232-233, or [8], p. 65) \((C)\int_{A}(f+c)d\mu=(C)\int_{A}fd\mu+c\cdot\mu(A)\).
+
+If \(\mu\) is submodular too, then for all \(f,g\) of arbitrary sign and lower bounded, the property of subadditivity holds (see, e.g., [8], p. 75, Theorem 6.3)
+
+\[(C)\int_{A}(f+g)d\mu\leq(C)\int_{A}fd\mu+(C)\int_{A}gd\mu.\]
+
+However, in particular, the comonotonic additivity holds, that is if \(\mu\) is a monotone set function and \(f,g\) are \(\mathcal{C}\)-measurable and comonotone on \(A\) (that is \((f(\omega)-f(\omega^{\prime}))\cdot(g(\omega)-g(\omega^{\prime}))\geq 0\), for all \(\omega,\omega^{\prime}\in A\)), then by, e.g., Proposition 5.1, (vi), p. 65 in [8], we have
+
+\[(C)\int_{A}(f+g)d\mu=(C)\int_{A}fd\mu+(C)\int_{A}gd\mu.\]
+
+(iii) If \(f\leq g\) on \(A\) then \((C)\int_{A}fd\mu\leq(C)\int_{A}gd\mu\) (see, e.g., [37], p. 228, Theorem 11.2, (3) if \(f,g\geq 0\) and p. 232 if \(f,g\) are of arbitrary sign).
+
+(iv) Let \(f\geq 0\). If \(A\subset B\) then \((C)\int_{A}fd\mu\leq(C)\int_{B}fd\mu.\) In addition, if \(\mu\) is finitely subadditive (that is, \(\mu(\bigcup_{k=1}^{n}A_{k})\leq\sum_{k=1}^{n}\mu(A_{k})\), for all \(n\in\mathbb{N}\)), then
+
+\[(C)\int_{A\cup B}fd\mu\leq(C)\int_{A}fd\mu+(C)\int_{B}fd\mu.\](v) It is immediate that \((C)\int_{A}1\cdot d\mu(t)=\mu(A)\).
+
+(vi) The formula \(\mu(A)=\gamma(m(A))\), where \(\gamma:[0,m(\Omega)]\to\mathbb{R}\) is an increasing and concave function, with \(\gamma(0)=0\) and \(m\) is a bounded measure (or bounded but only finitely additive) on a \(\sigma\)-algebra on \(\Omega\) (that is, \(m(\emptyset)=0\) and \(m\) is countably additive), gives simple examples of monotone and submodular set functions (see, e.g., [8], pp. 16-17). Such of set functions \(\mu\) are also called distorsions of countably additive measures (or distorted measures).
+
+If \(\Omega=[a,b]\), then for the Lebesgue (or any Borel) measure \(m\) on \([a,b]\), \(\mu(A)=\gamma(m(A))\) give simple examples of bounded, monotone and submodular set functions on \([a,b]\).
+
+In addition, if we suppose that \(\gamma\) is continuous at \(0\) and at \(m([a,b])\), then by the continuity of \(\gamma\) on the whole interval \([0,m([a,b])]\) and from the continuity from below of any Borel measure, it easily follows that the corresponding distorted measure also is continuous from below.
+
+For simple examples, we can take \(\gamma(t)=t^{p}\) with \(0<p<1\), \(\gamma(t)=\frac{2t}{1+t}\), \(\gamma(t)=1-e^{-t}\), \(\gamma(t)=\ln(1+t)\) for \(t\geq 0\) and \(\gamma(t)=\sin(t/2)\) for \(t\in[0,\pi]\).
+
+Now, let us consider that in the above definition of a distorted Lebesgue measure, \(\mu(A)=\gamma(m(A))\), in addition \(\gamma\) is considered strictly increasing and differentiable. In this case, if \(f\) is nonnegative, nondecreasing and continuous, then (see, e.g., [34], Theorem I)
+
+\[(C)\int_{0}^{x}f(s)d\mu(s)=\int_{0}^{x}\gamma^{\prime}(x-s)f(s)ds,\]
+
+while if \(f\) is nonnegative, nonincreasing and continuous, then (see, e.g., [34], Theorem A.1)
+
+\[(C)\int_{0}^{x}f(s)d\mu(s)=\int_{0}^{x}\gamma^{\prime}(s)f(s)ds.\]
+
+(vii) If \(\mu\) is a countably additive bounded measure, then the Choquet integral \((C)\int_{A}fd\mu\) reduces to the usual Lebesgue type integral (see, e.g., [8], p. 62, or [37], p. 226).
+
+(viii) Let \(\mathcal{C}\) be a \(\sigma\)-algebra of subsets in \([0,1]\) and \(\mu:\mathcal{C}\to[0,+\infty]\) be a monotone set function.The analogs of the Lebesgue spaces in the context of capacities can be introduced for \(1\leq p<+\infty\) via the formulas
+
+\[\mathcal{L}_{p,\mu}([0,1])=\{f:[0,1]\to\mathbb{R};f\text{ is }\mathcal{C} \text{- measurable }\text{ and }(C)\int_{0}^{1}|f(t)|^{p}\mathrm{d}\mu(t)<+\infty\}.\]
+
+When \(\mu\) is a subadditive capacity (in particular, when \(\mu\) is submodular), the functionals \(\|\cdot\|_{\mathcal{L}_{p,\mu}}\) given by
+
+\[\|f\|_{\mathcal{L}_{p,\mu}}=\left((C)\int_{\Omega}|f(t)|^{p}\mathrm{d}\mu(t) \right)^{1/p}\]
+
+satisfy the triangle inequality too (see, e.g. Theorem 2, p. 5 in [4] or Proposition 9.4, p. 109 in [8] or Theorems 3.5 and 3.7 in [36] or the comments in the proof of Theorem 3.4, Step 3 in [15].
+
+Denoting
+
+\[\mathcal{N}_{p}=\{f\in\mathcal{L}_{p,\mu}([0,1]);\left((C)\int_{0}^{1}|f(t)|^{p} \mathrm{d}\mu(t)\right)^{1/p}=0\},\]
+
+if \(\mu\) is a submodular capacity, then the functionals \(\|\cdot\|_{L_{p,\mu}([0,\) given by
+
+\[\|f\|_{L_{p,\mu}([0,1])}=\left((C)\int_{0}^{1}|f(t)|^{p}\mathrm{d}\mu(t)\right) ^{1/p}\]
+
+satisfy the axioms of a norm on the quotient space \(L_{p,\mu}([0,1])=\mathcal{L}_{p,\mu}([0,1])/\mathcal{N}_{p}\) (see [8], p. 109, Proposition 9.4 for \(p=1\) and p. 115 for arbitrary \(p\geq 1\)). If, in addition, \(\mu\) is continuous from below, then \(L_{p,\mu}([0,1])\) is a Banach space (see [8], pp. 11-12, Proposition 9.5) and \(h\in\mathcal{N}_{p}\) if and only if \(h=0\), \(\mu\)-a.e., meaning that there exists \(N\) with \(\mu^{*}(N)=0\), such that \(h(\omega)=0\), for all \(\omega\in\Omega\setminus N\) (see [8], p. 107, Corollary 9.2 and pp. 107-108 ). Here \(\mu^{*}\) is he outer measure attached to \(\mu\), given by the formula \(\mu^{*}(A)=\inf\{\mu(B);A\subset B,B\in\mathcal{C}\}\).
+
+Also, denote
+
+\[L_{p,\mu}^{+}([0,1])=\{f:[0,1]\rightarrow\mathbb{R};f\text{ is }\mathcal{C} \text{-measurable and }(C)\int_{0}^{1}|f(t)|^{p}d\mu(t)<+\infty\}.\]
+
+## 3 Preliminaries on nonlinear compact operators
+
+In this section we present a few well-known concepts and general results on compactness of nonlinear operators which we need for the next sections.
+
+**Definition 3.1.** Let \(A:X\to Y\) be a nonlinear operator between two metric spaces. We recall that \(A\) is said to be compact if it is continuous on \(X\) and for any bounded \(M\subset X\), \(A(M)\) is relatively compact in \(Y\) (that is the closure \(\overline{A(M)}\) is compact in \(Y\)). Recall here that a set \(M\subset X\) is called bounded in the metric space \((X,\rho)\) if \(diam(M)=\sup\{\rho(x,y);x,y\in M\}<+\infty\).
+
+**Remark 3.2** If \(X\) and \(Y\) are two normed spaces over \(\mathbb{R}\) and \(A\) is positive homogeneous, then it is easy to see that \(A\) is compact if and only if it is continuous and \(A[B_{1}]\) is relatively compact in \(Y\), where \(B_{r}=\{x\in X;\|x\|\leq r\}\). Indeed, let \(M\subset X\) be bounded, that is there exists \(r>0\), such that \(M\subset B_{r}\). It is immediate that \(\overline{A[M]}\subset\overline{A[B_{r}]}=r\overline{A(B_{1}]}\). Since \(r\overline{A[B_{1}]}\) is compact, it is clear that the closed set \(\overline{A[M]}\) is compact.
+
+We also recall that a fundamental result in the study of the algebra of continuous functions on a compact Hausdorff space is the well-known Arzela-Ascoli theorem, which can be stated as follows.
+
+**Theorem 3.3.** ([11], IV.6.7.) _Let \(X\) be a compact Hausdorff space and denote_
+
+\[C(X;\mathbb{R})=\{f:X\rightarrow\mathbb{R};f\text{ is continuous on }X\}.\]
+
+_Then a subset \(F\) of \(C(X;\mathbb{R})\) is relatively compact in the topology induced by the uniform norm, if and only if it is equicontinuous and pointwise bounded. Here pointwise bounded means that for any \(x\in X\) we have \(\sup\{|f(x)|;f\in F\}<\infty\)._Compactness of the Volterra-Choquet operators
+
+This section contains some important properties of compactness for the Volterra-Choquet operators.
+
+In this sense, firstly we need the following.
+
+**Theorem 4.1.**_Let \(\mu\) be a monotone, submodular and continuous from below set function on all Borelian subsets in \([0,1]\) (or all the Lebesgue measurable subsets in \([0,1]\)), \(1<p<+\infty\) and \(1/p+1/q=1\). Then, for all \(x,y\in[0,1]\) with \(x\leq y\), and \(f\in L^{+}_{p,\mu}([0,1])\), the Volterra Choquet operator \(V(f)(x)=(C)\int_{[0,x]}f(t)d\mu(t)\) has the property_
+
+\[|V(f)(x)-V(f)(y)|\leq\|f\|_{L_{p,\mu}([0,1])}\cdot\mu([x,y])^{1/q}.\]
+
+**Proof.** Indeed, without loss of generality, we may suppose that \(x<y\). The submodularity of \(\mu\) evidently implies the finitely subadditivity. Since \([0,y]=[0,x]\bigcup[x,y]\), by Remark 2.2, (iv), we obtain \(V(f)(y)\leq V(f)(x)+(C)\int_{x}^{y}f(t)d\mu(t)\), which by applying the Holder's inequality too ((see Remark 2.2, (viii)) e.g., implies
+
+\[0\leq V(f)(y)-V(f)(x)\leq(C)\int_{[x,y]}f(t)d\mu(t)=(C)\int_{[x,y]}f(t)\cdot 1 \cdot d\mu(t)\]
+
+\[\leq\|f\|_{L_{p,\mu}([0,1])}\cdot\left((C)\int_{[x,y]}1\cdot d\mu(t)\right)^{1 /q}=\|f\|_{L_{p,\mu}([0,1])}\cdot\mu([x,y])^{1/q}.\]
+
+If \(y\leq x\), then in the statement we obtain a similar inequality, by replacing \(\mu([x,y])\) with \(\mu([y,x])\), which proves the theorem. \(\square\)
+
+**Corollary 4.2.**_Suppose that \(\mu\) is a distorted Lebesgue measure, that is \(\mu(A)=\gamma(m(A))\), where \(m\) is the Lebesgue measure and \(\gamma:[0,1]\to\mathbb{R}\) is nondecreasing, concave, continuous on \([0,1]\) and \(\gamma(0)=0\). For \(1<p<+\infty\), denote \(B^{+}_{p,\mu,1}=\{f\in L^{+}_{p,\mu}([0,1]):\|f\|_{L_{p,\mu}([0,1])}\leq 1\}\). Then, \(V(B^{+}_{p,\mu,1})\) is an equicontinuous and uniformly bounded set of continuous functions defined on \([0,1]\)._
+
+**Proof.** Since by Remark 2.2, (vi), any distorted Lebesgue set function is submodular and continuous from below (in fact, from above too), by Theorem 4.1, we obtain
+
+\[|V(f)(x)-V(f)(y)|\leq\gamma(|x-y|)^{1/q},\mbox{ for all }x,y\in[0,1]\mbox{ and }f\in B^{+}_{p,\mu,1}([0,1]).\]
+
+Let \(\varepsilon>0\) be fixed. By the continuity of \(\gamma\), there exists a \(\delta>0\) (depending of course only on \(\varepsilon\) and \(\gamma\) and indepepndent of \(f\)), such that \(\gamma(|x-y|)^{1/q}<\varepsilon\), for all \(|x-y|<\delta\). This immediately implies the equicontinuity of the set of continuous functions \(V(B^{+}_{p,\mu,1})\).
+
+Also, choosing \(y=0\) in the above inequality, we obtain
+
+\[|V(f)(x)|\leq\gamma(x)^{1/q}\leq\gamma(1)^{1/q},\mbox{ for all }x\in[0,1],f\in B ^{+}_{p,\mu,1},\]which proves that \(V(B^{+}_{p,\mu,1})\) is uniformly bounded. 
+
+By using Definition 3.1, we can state the following.
+
+**Corollary 4.3.**_Under the hypothesis of Corollary 4.2, the Volterra-Choquet operator \(V:L^{+}_{p,\mu}([0,1])\to C_{+}[0,1]\subset C[0,1]\) is a nonlinear compact operator._
+
+_Here \(L^{+}_{p,\mu}([0,1])\) is endowed with the metric generated by the \(L_{p,\mu}\)-norm in Remark 2.2, (viii) and \(C_{+}[0,1]\) denotes the space of all nonnegative real-valued continuous functions on \([0,1]\), which is a metric space endowed with the metric generated by the uniform norm._
+
+**Proof.** By Arzela-Ascoli result in Theorem 3.3 and by Corollary 4.2, it follows that \(\overline{V(B^{+}_{p,\mu,1})}\) is compact.
+
+Let \(M\subset L^{+}_{p,\mu}([0,1])\) be bounded, that is \(d=diam(M)<+\infty\). For a fixed \(x_{0}\in M\) and an arbitrary \(x\in M\), we get
+
+\[|\quad\|x\|_{L_{p,\mu}([0,1])}-\|x_{0}\|_{L_{p,\mu}([0,1])}\quad|\leq\|x-x_{0} \|_{L_{p,\mu}([0,1])}\leq d,\]
+
+which immediately implies \(\|x\|_{L_{p,\mu}([0,1])}\leq\|x_{0}\|_{L_{p,\mu}([0,1])}+d\), that is
+
+\[M\subset B^{+}_{p,\mu,\|x_{0}\|_{L_{p,\mu}([0,1])}+d}=(\|x_{0}\|_{L_{p,\mu}([0,1])}+d)B^{+}_{p,\mu,1}.\]
+
+Applying \(V\) and taking into account that by Remark 2.2, (i), \(V\) is positive homogeneous, we get \(V(M)\subset(\|x_{0}\|_{L_{p,\mu}([0,1])}+d)V(B^{+}_{p,\mu,1})\), which implies \(\overline{V(M)}\subset(\|x_{0}\|_{L_{p,\mu}([0,1])}+d)\overline{V(B^{+}_{p, \mu,1})}\).
+
+Now, since in a metric space any closed subset of a compact set also is compact, it implies that \(\overline{V(M)}\) is compact.
+
+It remains to prove the continuity of the operator \(V:L^{+}_{p,\mu}([0,1])\to C_{+}[0,1]\). For this purpose, let \(f,g\in L_{p,\mu}([0,1])\) (in fact, not necessarily nonnegative). From Holder's inequality it is immediate that \(L^{+}_{p,\mu}([0,1])\subset L^{+}_{1,\mu}([0,1])\). Since according to Remark 2.2, (ii), the Choquet integral is subadditive, by \(f(s)\leq g(s)+|f(s)-g(s)|\), for all \(s\in[0,1]\), it follows that
+
+\[V(f)(t)\leq(C)\int_{0}^{t}g(s)d\mu(s)+(C)\int_{0}^{1}|f(s)-g(s)|d\mu(s).\]
+
+This implies \(V(f)(t)\leq V(g)(t)+V(|f-g|)(t)\), for all \(t\in[0,1]\).
+
+Also, by \(g(s)\leq f(s)+|g(s)-f(s)|\), for all \(s\in[0,1]\), by similar reasoning we obtain \(V(g)(t)\leq V(f)(t)+V(|f-g|)(t)\), for all \(t\in[0,1]\), which combined with the above inequality, leads to the inequality valid for all \(t\in[0,1]\)
+
+\[|V(f)(t)-V(g)(t)|\leq V(|f-g|)(t)\leq(C)\int_{0}^{1}|f(s)-g(s)|\cdot 1d\mu(s).\]
+
+Passing to supremum after \(t\in[0,1]\) in the left hand-side and then, applying the Holder's inequality to the right-hand side, we easily arrive to
+
+\[\|V(f)-V(g)\|_{C[0,1]}\leq\mu([0,1])^{1/q}\cdot\|f-g\|_{L_{p,\mu}([0,1])},\]
+
+from which easily follows the continuity of \(V\).
+
+Concluding, by Definition 3.1 all the above mean the compactness of \(V:L^{+}_{p,\mu}([0,1])\to C_{+}[0,1]\). \(\Box\)
+
+**Remark 4.4.** In the case of \(p=1\), Corollary 4.3 does not hold in general. Indeed, it is known that even in the very particular case when \(\gamma(t)=t\) (that is when \(\mu\) one reduces to the classical Lebesgue measure), the equicontinuity fails.
+
+## 5 Lipschitz type properties and compactness
+
+In this section, firstly we prove Lipschitz properties of the nonlinear Volterra-Choquet operator \(V\), on the whole spaces \(C[0,1]\) and \(L_{p,\mu}([0,1])\) with \(1\leq p<+\infty\).
+
+**Theorem 5.1.**_Let \(\mu\) be a monotone, submodular and continuous from below and from above set function on the class of all Borelian (or alternatively, on the class of all Lebesgue measurable) subsets of \([0,1]\)._
+
+_(i) If \(f\in L_{1,\mu}([0,1])\) then \(V(f)\in L_{1,\mu}([0,1])\) and for all \(f,g\in L_{1,\mu}([0,1])\) we have_
+
+\[\|V(f)-V(g)\|_{L_{1,\mu}([0,1])}\leq\mu([0,1])\cdot\|f-g\|_{L_{1,\mu}([0,1])}.\]
+
+_(ii) If \(f\in C[0,1]\) then \(V(f)\in C[0,1]\) and for all \(f,g\in C[0,1]\) we have_
+
+\[\|V(f)-V(g)\|_{C[0,1]}\leq\mu([0,1])\cdot\|f-g\|_{C[0,1]},\]
+
+_where \(\|\cdot\|_{C[0,1]}\) denotes the uniform norm on \(C[0,1]\)._
+
+_(iii) Let \(1<p<+\infty\). If \(f\in L_{p,\mu}([0,1])\) then \(V(f)\in L_{p,\mu}([0,1])\) and for all \(f,g\in L_{p,\mu}([0,1])\) we have_
+
+\[\|V(f)-V(g)\|_{L_{p,\mu}([0,1])}\leq\mu([0,1])\cdot\|f-g\|_{L_{p,\mu}([0,1])}.\]
+
+**Proof.** (i) Firstly, we need to show that if \(f\in L_{1,\mu}([0,1])\), then \(V(f)\in L_{1,\mu}([0,1])\). In the case when \(f\geq 0\), the proof is simple, because \(V(f)(x)=(C)\int_{0}^{x}f(t)d\mu(t)\) is evidently a nondecreasing function on \([0,1]\). Therefore, \(V(f)(x)\) is a Borel (Lebesgue) measurable function.
+
+Suppose now that \(f\) is bounded and has negative values too, that is there exist \(M^{\prime}<0\) and \(M>0\), such that \(M^{\prime}\leq f(t)\leq M\), \(t\in[0,1]\). By Definition 2.1, (ii), we have
+
+\[V(f)(x)\] \[=\int_{0}^{+\infty}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha+\int_{- \infty}^{0}\left[\mu(\{t\in[0,x];f(t)\geq\alpha\})-\mu([0,x])\right]d\alpha\] \[=\int_{0}^{M}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha+\int_{M^{ \prime}}^{0}\left[\mu(\{t\in[0,x];f(t)\geq\alpha\})-\mu([0,x])\right]d\alpha\] \[=\int_{0}^{M}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha+\int_{M^{ \prime}}^{0}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha+M^{\prime}\mu([0,x]). \tag{3}\]Therefore, \(V(f)(x)\) is the sum of two nondecreasing functions with a nonincreasing one, all of them being Borel (Lebesgue) measurable, implying that \(V(f)(x)\) is Borel (Lebesgue) measurable too.
+
+Suppose now that \(f\) is unbounded and has negative values too. By the above formula, we can write \(V(f)(x)=F(x)+G(x)\), with
+
+\[F(x)=\int_{0}^{+\infty}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha,\]
+
+\[G(x)=\int_{-\infty}^{0}[\mu(\{t\in[0,x];f(t)\geq\alpha\})-\mu([0,x])]d\alpha\]
+
+\[=\lim_{n\rightarrow+\infty}\int_{-n}^{0}[\mu(\{t\in[0,x];f(t)\geq\alpha\})- \mu([0,x])]d\alpha.\]
+
+Evidently \(F(x)\) is nondecreasing and therefore Borel (Lebesgue) measurable. Then, since for each \(n\in\mathbb{N}\),
+
+\[\int_{-n}^{0}[\mu(\{t\in[0,x];f(t)\geq\alpha\})-\mu([0,x])]d\alpha\]
+
+\[=\int_{-n}^{0}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha-n\mu([0,x])\]
+
+is Borel (Lebesgue) measurable as a difference of two measurable functions, it follows that \(G(x)\) is Borel (Lebesgue) measurable as a pointwise limit of Borel (Lebesgue) measurable functions.
+
+Concluding, in this case too we have that \(V(f)(x)\) is Borel (Lebesgue) measurable.
+
+Then, we have
+
+\[(C)\int_{0}^{1}|V(f)(x)|d\mu(x)=(C)\int_{0}^{1}\left|(C)\int_{0}^{x}f(t)d\mu( t)\right|d\mu(x)\]
+
+\[\leq(C)\int_{0}^{1}\left[(C)\int_{0}^{1}|f(t)|d\mu(t)\right]d\mu(x)=\|f\|_{L_{ 1,\mu}([0,1])}\cdot(C)\int_{0}^{1}d\mu(x)\]
+
+\[=\mu([0,1])\cdot\|f\|_{L_{1,\mu}([0,1])}<+\infty.\]
+
+Let \(f,g\in L_{1,\mu}([0,1])\). Since according to Remark 2.2, (ii), the Choquet integral is subadditive, by \(f(s)\leq g(s)+|f(s)-g(s)|\), for all \(s\in[0,1]\), it follows that
+
+\[V(f)(t)=(C)\int_{0}^{t}f(s)d\mu(s)\leq(C)\int_{0}^{t}g(s)d\mu(s)+(C)\int_{0}^{ 1}|f(s)-g(s)|d\mu(s),\]
+
+which implies \(V(f)(t)\leq V(g)(t)+V(|f-g|)(t)\), for all \(t\in[0,1]\).
+
+[MISSING_PAGE_FAIL:10]
+
+The Lipschitz inequality, follows immediately by passing to supremum after \(t\in[0,1]\) in formula (4).
+
+(iii) Let \(f\in L_{p,\mu}([0,1])\) with \(1<p<+\infty\). Since from Holder's inequality we get \(L_{p,\mu}([0,1])\subset L_{1,\mu}([0,1])\), reasoning as at the point (i), it follows that \(V(f)(x)\) is Borel (Lebesgue) measurable.
+
+By (4) and using the Holder's inequality (see Remark 2.2, (viii)), it follows
+
+\[|V(f)(t)-V(g)(t)|\leq\int_{0}^{1}|f(s)-g(s)|\cdot 1\cdot d\mu(s)\]
+
+\[\leq\left((C)\int_{0}^{1}|f(s)-g(s)|^{p}d\mu(s)\right)^{1/p}\cdot\mu([0,1])^{1 /q}.\]
+
+Taking to the power \(p\) both members of the above inequality, applying the Choquet ntegral on \([0,1]\) with respect to \(t\) and then taking to the power \(1/p\), we obtain
+
+\[\|V(f)-V(g)\|_{L_{p,\mu}([0,1])}\leq\mu([0,1])^{1/p}\cdot\mu([0,1])^{1/q}\cdot \|f-g\|_{L_{p,\mu}([0,1])}\]
+
+\[=\mu([0,1])\cdot\|f-g\|_{L_{p,\mu}([0,1])},\]
+
+which ends the proof of the theorem. \(\square\)
+
+**Remark 5.2.** Any distorted Lebesgue measure (defined as in Remark 2.2, (vi)) satisfies the hypothesis in Theorem 5.1.
+
+**Remark 5.3.** By Theorem 5.1, (i) and (iii), for the norm of \(V\) given by
+
+\[\||V\||_{L_{p,\mu}([0,1])}=\sup\left\{\frac{\|V(f)-V(g)\|_{L_{p,\mu}([0,1])}}{ \|f-g\|_{L_{p,\mu}([0,1])}};f,g\in L_{p,\mu}([0,1]),f\neq g\right\}\]
+
+we have \(\||V\||_{L_{p,\mu}([0,1])}\leq\mu([0,1])\), \(1\leq p<+\infty\).
+
+As an application of Theorem 5.1, (ii), the following compactness property holds.
+
+**Corollary 5.4.**_Suppose that \(\mu\) is a distorted Lebesgue measure, that is \(\mu(A)=\gamma(m(A))\), where \(m\) is the Lebesgue measure, \(\gamma:[0,1]\to\mathbb{R}\) is non-decreasing, concave, continuous on \([0,1]\) and \(\gamma(0)=0\). Then, the Volterra-Choquet operator \(V:C[0,1]\to C[0,1]\) defined by \(V(f)(x)=(C)\int_{0}^{x}f(t)d\mu(t)\) is a nonlinear compact operator._
+
+**Proof.** By Theorem 5.1, (ii), we have \(V:C[0,1]\to C[0,1]\) and obviously that the Lipschitz property implies the continuity of \(V\).
+
+Let \(B_{\mu,1}=\{f\in C[0,1];\|f\|_{C[0,1]}\leq 1\}\). If we prove that \(V(B_{\mu,1})\) is equicontinuous and pointwise bounded in \(C[0,1]\), then since by Remark 2.2, (i) \(V\) is positive homogeneous, applying Remark 3.2 and Theorem 3.3, we will immediately obtain that \(V\) is compact.
+
+Indeed, firstly for \(f\in B_{\mu,1}\) we get
+
+\[|V(f)(x)|\]
+
+\[=|(C)\int_{[0,x]}f(t)|d\mu(t)|\leq\|f\|_{C[0,1]}\cdot\gamma(m([0,1]))=\gamma( 1),\mbox{ for all }x\in[0,1]\]which implies \(\|V(f)\|_{C[0,1]}\leq\gamma(1)\), for all \(f\in B_{\mu,1}\). This means that \(V(B_{\mu,1})\) is uniformly bounded (more than pointwise bounded).
+
+Now, applying (3) for \(M^{\prime}=-1\), \(M=+1\), for all \(f\in B_{\mu,1}\) and \(x\in[0,1]\), we get
+
+\[V(f)(x)=\]
+
+\[=\int_{0}^{1}\mu(\{t\in[0,x];f(t)\geq\alpha\})d\alpha+\int_{-1}^{0}\mu(\{t\in[ 0,x];f(t)\geq\alpha\})d\alpha-\mu([0,x]).\]
+
+Let \(x_{n}\to x\). We easily get
+
+\[|V(f)(x_{n})-V(f)(x)|\]
+
+\[\leq\int_{0}^{1}|\mu(\{t\in[0,x_{n}];f(t)\geq\alpha\})-\mu(\{t\in[0,x];f(t)\geq \alpha\})|d\alpha\]
+
+\[+\int_{-1}^{0}|\mu(\{t\in[0,x_{n}];f(t)\geq\alpha\})-\mu(\{t\in[0,x];f(t)\geq \alpha\})|d\alpha\]
+
+\[+|\mu([0,x_{n}])-\mu([0,x])|.\]
+
+Firstly, suppose that \(x_{n}\searrow x\). Since \([0,x_{n}]=[0,x]\cup[x,x_{n}]\) and \(\mu\) submodular implies that \(\mu\) is subadditive, we obtain
+
+\[0\leq\mu(\{t\in[0,x_{n}];f(t)\geq\alpha\})-\mu(\{t\in[0,x];f(t)\geq\alpha\})\]
+
+\[\leq\mu(\{t\in[x,x_{n}];f(t)\geq\alpha\})\leq\mu([x,x_{n}]),\]
+
+which immediately implies
+
+\[|V(f)(x_{n})-V(f)(x)|\leq 3\mu([x,x_{n}]).\]
+
+The continuity of \(\mu\) from above, immediately implies that \(\lim_{n\to\infty}|V(f)(x_{n})-V(f)(x)|=0\), independent of \(f\).
+
+If \(x_{n}\nearrow x\) we write \([0,x]=[0,x_{n}]\cup[x_{n},x]\) and by analogous reasonings (since \(\mu\) is continuous from below too) we get
+
+\[|V(f)(x_{n})-V(f)(x)|\leq 3\mu([x_{n},x]).\]
+
+Concluding, \(V(B_{\mu,1})\) is equicontinuous and therefore Corollary 5.4 is proved. \(\Box\)
+
+## 6 Cyclicity
+
+Firstly recall the following known concepts.
+
+**Definition 6.1.** Let \((X,\|\cdot\|)\) be a Banach space on \(K=\mathbb{R}\) (the real line) or \(\mathbb{C}\) (the complex plane).
+
+(i) The (not necessarily linear) continuous operator \(T:X\to X\) is called cyclic, if there exists \(x\in X\) such that the linear span of \(Orb(T,x)\) is dense in \(X\). Here \(Orb(T,x)=\{x,T(x),T^{2}(x),...,T^{n}(x),...,\}\).
+
+(ii) \(T\) is called hypercyclic, if there exists \(x\in X\), such that the orbit \(Orb(T,x)\) is dense in \(X\). Of course, if \(X\) supports such an operator, then \(X\) must be separable.
+
+(iii) \(T\) is called supercyclic if there exists \(x\in X\) such that the set \(\mathcal{M}(x)=\{\lambda y;y\in Orb(T,x);\lambda\in K\}\) is dense in \(X\).
+
+In the classical case, it is well-known that the Volterra operator \(O\) given by (1) and the identity plus Volterra operator, \(I+O\), are cyclic operators on \(L^{2}[0,1]\), but they cannot be supercyclic and hypercyclic, see, e.g., [27] and [20].
+
+In what follows, we deal with cyclic type properties of the Volterra-Choquet operators with respect to a particular distorted Lebesgue measure. The problem of cyclic properties in the most general case remains open.
+
+**Theorem 6.2.**_Suppose that \(\mu\) is the distorted Lebesgue measure given by \(\mu(A)=\gamma(m(A))\), where \(m\) is the Lebesgue measure, \(\gamma:[0,1]\rightarrow\mathbb{R}\) is \(\gamma(x)=1-e^{-x}\) and that \(f_{0}(x)=1\), for all \(x\in[0,1]\). Then, for the Volterra-Choquet operator \(V\) with respect to \(\mu\), we have_
+
+\[Orb(V,f_{0})=\left\{1,\left\{1-e^{-x}\sum_{k=0}^{n-1}\frac{x^{k}}{k!},n=1,2,...,\right\}\right\}.\]
+
+**Proof.** Firstly, by direct calculation we get
+
+\[V(f_{0})(x)=\int_{0}^{\infty}\mu(\{s\in[0,x];1\geq\alpha\})d\alpha=\int_{0}^{ 1}\mu(\{s\in[0,x];1\geq\alpha\})d\alpha\]
+
+\[=\mu([0,x])=\gamma(m([0,x]))=\gamma(x)=1-e^{-x}.\]
+
+But, according to Proposition 1 in [34], see also Remark 2.2, (vi), (since \(\gamma(x)\) is strictly increasing) and nonnegative), it follows
+
+\[V^{2}(f_{0})(x)\]
+
+\[=(C)\int_{0}^{x}V(f_{0})(s)d\mu(s)=\int_{0}^{x}\gamma^{\prime}(x-s)\gamma(s)ds\]
+
+\[=\int_{0}^{x}e^{-(x-s)}(1-e^{-s})ds=1-e^{-x}-xe^{-x}=V(f_{0})(x)-xe^{-x}:=g_{ 1}(x).\]
+
+Since \(g_{1}\) is strictly increasing and nonnegative on \([0,1]\) (\(g_{1}(0)=0\)), again by Proposition 1 in [34], it follows
+
+\[V^{3}(f_{0})(x)=\int_{0}^{x}\gamma^{\prime}(x-s)V^{2}(f_{0})(s)ds=\int_{0}^{ x}e^{-(x-s)}[1-e^{-s}-se^{-s}]ds\]
+
+\[=e^{-x}\int_{0}^{x}(e^{s}-1-s)ds=e^{-x}(e^{x}-1-x-\frac{x^{2}}{2})=1-e^{-x}-xe ^{-x}-e^{-x}\cdot\frac{x^{2}}{2}.\]
+
+\[=V^{2}(f_{0})(x)-e^{-x}\cdot\frac{x^{2}}{2}:=g_{2}(x).\]Since \(g_{2}\) is strictly increasing and nonnegative on \([0,1]\), we get
+
+\[V^{4}(f_{0})(x)=\int_{0}^{x}\gamma^{\prime}(x-s)V^{3}(f_{0})(s)ds= \int_{0}^{x}e^{-(x-s)}[V^{2}(f_{0})(s)-e^{-s}\cdot\frac{s^{2}}{2}]ds\] \[=\int_{0}^{x}e^{-(x-s)}V^{2}(f_{0})(s)ds-e^{-x}\int_{0}^{x}\frac{s ^{2}}{2}ds=V^{2}(f_{0})(x)-e^{-x}\cdot\frac{x^{2}}{2}-e^{-x}\cdot\frac{x^{3}}{ 3!}\] \[=V^{3}(f_{0})(x)-e^{-x}\cdot\frac{x^{3}}{3!}.\]
+
+Continuing this kind of reasoning, we easily arrive at the general recurrence formula
+
+\[V^{n}(f_{0})(x)=V^{n-1}(f_{0})(x)-e^{-x}\cdot\frac{x^{n-1}}{(n-1)!},\]
+
+for all \(n\geq 2\), which proves the theorem. \(\square\)
+
+**Corollary 6.3.**_Let \(V\) be the Volterra-Choquet operator with respect to the distorted Lebesgue measure \(\mu\) in Theorem 6.2._
+
+_(i) As mapping \(V:C[0,1]\to C[0,1]\), \(V\) is a cyclic operator, with respect to the density induced by the uniform norm ;_
+
+_(ii) Also, as mapping \(V:L_{p,\mu}([0,1])\to L_{p,\mu}([0,1])\), \(1\leq p<+\infty\), \(V\) is a cyclic operator with respect to the density induced by the \(\|\cdot\|_{L_{p,\mu}([0,1])}\)-norm._
+
+**Proof.** (i) By Theorem 6.2, it is immediate that span \(Orb(V,f_{0})\) contains the countable subset given by \(\left\{1,e^{-x}\cdot\frac{x^{n}}{n!},n=0,1,2,....\right\}\), whose linear span is evidently dense in \(C[0,1]\) due to the Weierstrass approximation theorem by uniformly convergent sequences of polynomials.
+
+(ii) It suffices to show that the set of polynomials is dense in \(L_{p,\mu}([0,1])\) with respect to the norm \(\|\cdot\|_{L_{p,\mu}([0,1])}\). Indeed, since \(\gamma^{\prime}(0)<+\infty\), by Remark 3.3. and Corollary 3.4 in [15], for the Bernstein-Durrmeyer-Choquet polynomials denoted by \(D_{n,\mu}(f)(x)\) one has
+
+\[\|D_{n,\mu}(f)-f\|_{L_{p,\mu}([0,1])}\leq c_{0}K(f;1/\sqrt{n})_{L_{p,\mu}},\]
+
+where \(C^{1}_{+}([0,1])\) denotes the the space of all differentiable, nonnegative functions with \(g^{\prime}\) bounded on \([0,1]\) and \(K(f;t)_{L_{p,\mu}}=\inf_{g\in C^{1}_{+}([0,1])}\{\|f-g\|_{L_{p,\mu}([0,1])}+t\| g^{\prime}\|_{C[0,1]}\}\). According to Remark 3.3 in [15], it will suffices to prove that \(\lim_{t\to 0}K(f;t)_{L_{p,\mu}}=0\), for all \(f\geq 0\). Indeed, by e.g., [22], there exist a sequence of non negative polynomials \((P_{n})_{n\in\mathbb{N}}\), such that \(\|f-P_{n}\|_{p}\to 0\) as \(n\to\infty\). For arbitrary \(\varepsilon>0\), let \(P_{m}\) be such \(\|f-P_{m}\|_{p}<\varepsilon/2\). Then for all \(t\in(0,\,\varepsilon/(2\|P_{m}^{\prime}\|_{\infty}))\), we get
+
+\[K(f;t)_{p}\ \leq\ \|f-P_{m}\|_{p}\ +\ t\|P_{m}^{\prime}\|_{\infty}\ <\ \varepsilon/2\ +\ \varepsilon/2\ =\ \varepsilon,\]
+
+which proves our assertion. \(\square\)
+
+**Remark 6.4.** Using similar calculations with those in the proof of Theorem 6.2, we easily obtain that the operator \(U=I+V\) satisfies the cyclicity properties in Corollary 6.3.
+
+**Remark 6.5**.: The question that the Volterra-Choquet operator \(V\) in Theorem 6.2 is, or is not, hypercyclic or supercyclic remains unsettled. We observe that for \(f_{0}=1\), neither \(Orb(V,f_{0})\) and nor \(\mathcal{M}(f_{0})\) are not dense in \(C[0,1]\).
+
+**Declaration of interest :** None.
+
+## References
+
+* [1] D. R. Adams, Choquet integrals in potential theory, Publicacions Matematiques, **42** (1998), 3-66.
+* [2] O. Baksi, T. Khan, J. Lang and V. Musil, Strict S-numbers of the Volterra operator, Proc. Amer. Math. Soc., **146** (2018), no. 2, 723-731.
+* [3] M. S. Brodskii, On a problem of I.M. Gelfand, Uspekhi Mat. Nak., **12** (1957), 129- 132.
+* [4] J. Cerda, J., Martin and P. Silvestre, Capacitary function spaces, Collect. Math., **62** (2011), 95-118.
+* [5] G. Choquet, Theory of capacities, Annales de l' Institut Fourier, **5** (1954), 131-295.
+* [6] G. Choquet, La naissance de la theorie des capacites: reflexion sur une experience personnelle, Comptes rendus de l'Academie des sciences. Serie generale, La Vie des sciences, **3** (1986), 385-397.
+* [7] J. B. Conway, A Course in Operator Theory, Rhode Island, Amer. Math. Soc., 2000.
+* [8] D. Denneberg, Non-Additive Measure and Integral, Kluwer Academic Publisher, Dordrecht, 1994.
+* [9] J. Dixmier, Les operateurs permutables a l'operateur integral, Portugal Math., **8** (1949), 73-84.
+* [10] W. F. Donoghue, The lattice of invariant subspaces of quasi-nilpotent completely continuous transformation, Pacific J. Math., **7** (1957), 1031-1035.
+* [11] N. Dunford and J. T. Schwartz, Linear Operators, vol. **1**, Wiley-Interscience, 1958.
+* [12] A. F. M. ter Elst and J. Zemanek, Contractive polynomials of the Volterra operator, Studia Math., **240** (2018), no. 3, 201-211.
+* [13] S. G. Gal and B. D. Opris, Uniform and pointwise convergence of Bernstein-Durrmeyer operators with respect to monotone and submodular set functions, J. Math. Anal. Appl., **424** (2015), 1374-1379.
+* [14] S. G. Gal, Approximation by Choquet integral operators, Ann. Mat. Pura Appl., **195** (2016), no. 3, 881-896.
+
+* [15] S. G. Gal and S. Trifa, Quantitative estimates in \(L^{p}\)-approximation by Bernstein-Durrmeyer-Choquet operators with respect to distorted Borel measures, Results Math., **72** (2017), no. 3, 1405-1415.
+* [16] S. G. Gal, Uniform and pointwise quantitative approximation by Kantorovich-Choquet type integral operators with respect to monotone and submodular set functions, Mediterr. J. Math., **14** (2017), no. 5, art. 205, 12 pp.
+* [17] S. G. Gal, Quantitative approximation by nonlinear Picard-Choquet, Gauss-Weierstrass-Choquet and Poisson-Cauchy-Choquet singular integrals, Results Math., **73** (2018), no. 3, art. 92, 23 pp.
+* [18] S. G. Gal, Fredholm-Choquet integral equations, J. Integral Equations and Applications, [https://projecteuclid.org/euclid.jiea/1542358961](https://projecteuclid.org/euclid.jiea/1542358961) (under press).
+* [19] S. G. Gal, Volterra-Choquet integral equations, J. Integral Equations and Applications, [https://projecteuclid.org/euclid.jiea/1541668067](https://projecteuclid.org/euclid.jiea/1541668067) (under press).
+* [20] E. Gallardo and A. Montes, Volterra operator is not supercyclic, Integral Equations and Operator Theory, **50** (2004), no. 2, 211-216.
+* [21] P. R. Halmos, A Hilbert Space Problem Book, New York, Springer Verlag, 1967.
+* [22] K. G. Ivanov, On a new characteristic of functions, II. Direct and converse theorems for the best algebraic approximation in \(C[-1,1]\) and \(L^{p}[-1,1]\), PLISKA Stud. Math. Bulgar., **5**, (1983), 151-163.
+* [23] G. K. Kalisch, On similarity, reducing manifolds, and unitary equivalence of certain Volterra operators, Ann. Math., **66** (1957), 481-494.
+* [24] L. Khadkhuu and D. Tsedenbayar, A note about Volterra operator. Math. Slovaca, **68** (2018), no. 5, 1117-1120.
+* [25] L. Khadkhuu and D. Tsedenbayar, On the numerical range and numerical radius of the Volterra operator, The Bulletin of Irkutsk State University. Series Mathematics, **24** (2018), 102-108.
+* [26] P. Lefevre, The Volterra operator is finitely strictly singular from \(L^{1}\) to \(L_{\infty}\), J. Approx. Theory **214** (2017), 1-8.
+* [27] F. Leon-Saavedra and A. Piqueras-Lerena, Cyclic properties of Volterra operator, Pacific J. Math., **211** (2003), no. 1, 157-162.
+* [28] F. Leon-Saavedra and A. Piqueras-Lerena, Super convex-cyclicity and the Volterra operator, Monatsh. Math., **177** (2015), no. 2, 301-305.
+
+* [29] Yu-Xia, Liang and Rongwei, Yang, Energy functional of the Volterra operator, Banach J. Math. Anal., **13** (2019), no. 2, 255-274.
+* [30] G. Little and J. B. Reade, Estimates for the norm of the n-th indefinite integral, Bull. London Math. Soc., **30** (1998), 539-542.
+* [31] N. H. Salas, Supercyclicity and weighted shifts, Studia Math., **135** (1999), 55-74.
+* [32] D. Sarason, A remark on the Volterra operator, J. Math. Anal. Appl., **12** (1965), 244-246.
+* [33] Gh. Siretchi, Differential and Integral Calculus, vol. II, Edit. Stiint. Enciclop., Bucharest, 1985.
+* [34] M. Sugeno, A note on derivatives of functions with respect to fuzzy measures, Fuzzy Sets Syst., **222** (2013), 1-17.
+* [35] D. Tsedenbayar, On the power boundedness of certain Volterra operator pencils, Studia Math., **156** (2003), 59-66.
+* [36] R. S. Wang, Some inequalities and convergence theorems for Choquet integrals, J. Appl. Math. Comput., **35**(2011), 305-321.
+* [37] Z. Wang and G. J. Klir, Generalized Measure Theory, Springer, New York, 2009.
+* [38] Z. Wang and J.-A. Yan, A selective overview of applications of Choquet integrals, Advanced Lectures in Mathematics, pp. 484-515, Springer, 2007.

2003.00005v1.mmd

@@ -0,0 +1,297 @@
+# Elliptic Solutions for Higher Order KdV Equations
+
+ Masahito Hayashi
+
+Osaka Institute of Technology, Osaka 535-8585, Japan
+
+Kazuyasu Shigemoto
+
+Tezukayama University, Nara 631-8501, Japan
+
+Takuya Tsukioka
+
+Bukkyo University, Kyoto 603-8301, Japan
+
+masahito.hayashi@oit.ac.jpshigemot@tezukayama-u.ac.jptsukioka@bukkyo-u.ac.jp
+
+###### Abstract
+
+We study higher order KdV equations from the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group point of view. We find elliptic solutions of higher order KdV equations up to the ninth order. We argue that the main structure of the trigonometric/hyperbolic/elliptic \(N\)-soliton solutions for higher order KdV equations is the same as that of the original KdV equation. Pointing out that the difference is only the time dependence, we find \(N\)-soliton solutions of higher order KdV equations can be constructed from those of the original KdV equation by properly replacing the time-dependence. We discuss that there always exist elliptic solutions for all higher order KdV equations.
+
+## 1 Introduction
+
+The soliton system is taken an interest in for a long time by considering that the soliton equation is the concrete example of the exactly solvable nonlinear differential equation [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. Nonlinear differential equation relates to the interesting non-perturbative phenomena, so that studies of the soliton system are important to unveil mechanisms of various interesting physical phenomena such as those in superstring theories. It is quite surprising that such nonlinear soliton equations can be exactly solvable and have \(N\)-soliton solutions. Then we have a dogma that there must be the Lie group structure behind the soliton system, which is a key stone to make nonlinear differential equations exactly solvable.
+
+For the KdV soliton system, the Lie group structure is implicitly built in the Lax operator \(L=\partial_{x}^{2}-u(x,t)\). In order to see the Lie group structure, it is appropriate to formulate by using the linear differential operator \(\partial_{x}\) as the Schrodinger representation of the Lie algebra, which naturally comes to use the AKNS formalism [4] for the Lax equation
+
+\[\frac{\partial}{\partial x}\left(\begin{array}{c}\psi_{1}(x,t)\\ \psi_{2}(x,t)\end{array}\right)=\left(\begin{array}{cc}a/2&-u(x,t)\\ -1&-a/2\end{array}\right)\left(\begin{array}{c}\psi_{1}(x,t)\\ \psi_{2}(x,t)\end{array}\right).\]
+
+Then the Lie group becomes GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) for the KdV equation. An addition formula for elements of this Lie group is the well-known KdV type Backlund transformation.
+
+In our previous papers [13, 14, 15, 16], we have studied GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group approach for the unified soliton systems of KdV/mKdV/sinh-Gordon equations. Using the well-know KdV type Backlund transformation as the addition formula, we have algebraically constructed \(N\)-soliton solutions from various trigonometric/hyperbolic 1-soliton solutions [13, 15, 16]. Since the Lie group structure of KdV equation is the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1), which has elliptic solution, we expect that elliptic \(N\)-soliton solutions for the KdV equation can be constructed by using the Backlund transformation as the addition formula. We then really have succeeded in constructing elliptic \(N\)-soliton solutions [14].
+
+We can interpret this fact in the following way: The KdV equation, which is a typical 2-dimensional soliton equation, has the SO(2,1) Lie group structure and the well-known KdV type Backlund transformation can be interpreted as the addition formula of this Lie group. Then the elliptic function appears as a representation of the Backlund transformation. While, 2-dimensional Ising model, which is a typical 2-dimensional statistical integrable model, has the SO(3) Lie group structure and the Yang-Baxter relation can be interpreted as the addition formula of this Lie group. Then the elliptic function appears as a representation of the Yang-Baxter relation, which is equivalent to the addition formula of the spherical trigonometry [17, 18]. In 2-dimensional integrable, soliton, and statistical models, there is the SO(2,1)/SO(3) Lie group structure behind the model. As representations of the addition formula, the Backlund transformation, and the Yang-Baxter relation, there appears an algebraic function such as the trigonometric/hyperbolic/elliptic functions, which is the key stone to make the 2-dimensional integrable model into the exactly solvable model.
+
+In this paper, we consider Lax type higher order KdV equations and study trigonometric/hyperbolic/elliptic solutions. So far special hyperelliptic solutions for more than the fifth order KdV equation have been vigorously studied by formulating it into the Jacobi's inversion problem [19, 20, 21, 22, 23, 24]. Since the Lie group structure GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) and the Backlund transformation are common even for higher order KdV equations, we expect that there always exist elliptic solutions even for higher order. Then we study to find elliptic solutions up to the ninth order KdV equation, instead of special hyperelliptic solutions. We would like to conclude that we always have elliptic solutions for all higher order KdV equations.
+
+The paper is organized as follows: In section 2, we study trigonometric/hyperbolic solutions for higher order KdV equations. We construct elliptic solutions for higher order KdV equations in section 3. In section 4, we consider the KdV type Backlund transformation as an addition formula for solutions of the Weierstrass type elliptic differential equation. In section 5, we study special 1-variable hyperelliptic solutions, and we discuss a relation between such special 1-variable hyperelliptic solutions and our elliptic solutions. We devote a final section to summarize this paper and to give discussions.
+
+## 2 Trigonometric/hyperbolic solutions for the Lax type higher order KdV equations
+
+Lax pair equations for higher order KdV equations are given by
+
+\[L\psi=\frac{a^{2}}{4}\psi, \tag{2.1}\] \[\frac{\partial\psi}{\partial t_{2n+1}}=B_{2n+1}\psi, \tag{2.2}\]
+
+where \(L=\partial_{x}^{2}-u\). By using the pseudo-differential operator \(\partial_{x}^{-1}\), \(B_{2n+1}\) are constructed from \(L\) in the form [25, 26]
+
+\[B_{2n+1}=\left(\mathcal{L}^{2n+1}\right)_{\geq 0}=\partial_{x}^{2n+1}-\frac{2n+ 1}{2}u\partial_{x}^{2n-1}+\cdots, \tag{2.3}\]with
+
+\[{\cal L}=L^{1/2}=\partial_{x}-\frac{u}{2}\partial_{x}^{-1}+\frac{u_{x}}{4} \partial_{x}^{-2}+\cdots,\]
+
+where we denote "\(\geq 0\)" to take positive differential operator parts or function parts for general pseudo-differential operators. The integrability condition gives higher order KdV equations
+
+\[\frac{\partial L}{\partial t_{2n+1}}=[B_{2n+1},\,L]. \tag{2.4}\]
+
+As these higher order KdV equations comes from the Lax formalism, these higher order KdV equations are called the Lax type. There are various higher order KdV equations such as the Sawada-Kotera type, which is the higher order generalization of the Hirota form KdV equation [27]. As operators \(B_{2n+1}\) are constructed from \(L\), higher order KdV equations also have the same Lie group structure GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) as that of the original KdV(=third order KdV) equation. Using \(u=z_{x}\), the KdV type Backlund transformation is given in the form
+
+\[z_{x}^{\prime}+z_{x}=-\frac{a^{2}}{2}+\frac{(z^{\prime}-z)^{2}}{2}, \tag{2.5}\]
+
+which comes from Eq.(2.1) only, so that it is valid even for the higher order KdV equations. In the Lie group approach to the soliton system, if we find 1-soliton solutions, we can construct \(N\)-soliton solutions from various 1-soliton solutions by the Backlund transformation Eq.(2.5) as an addition formula of the Lie group.
+
+For 1-soliton solution of Eq.(2.4), if \(x\) and \(t_{2n+1}\) come in the combination \(X^{(2n+1)}=\alpha x+\beta t_{2n+1}^{\gamma}+\delta\), then if \(\gamma\neq 1\), the right-hand side of Eq.(2.4) is a function of only \(X\), while the left-hand side is a function of \(X\) and \(t\). Therefore, \(\gamma=1\) is necessary, that is, \(X=\alpha x+\beta t_{2n+1}+\delta\). \(N\)-soliton solutions are constructed from various 1-soliton solutions by the Backlund transformation. Then the main structure of \(N\)-soliton solutions, which are expressed with \(X_{i}^{(2n+1)},(i=1,2,\cdots,N)\), takes the same functional forms in higher order KdV equations and in the original KdV equation. The difference is only the time dependence of \(X_{i}=\alpha_{i}x+\beta_{i}t_{2n+1}+\delta_{i},(i=1,2,\cdots,N)\), that is, coefficients \(\beta_{i}\). This is valid not only for the trigonometric/hyperbolic \(N\)-soliton solutions but also for elliptic \(N\)-soliton solutions.
+
+For the trigonometric/hyperbolic \(N\)-soliton solutions, we can easily determine the time dependence without knowing details of \(B_{2n+1}\). For dimensional analysis, we have \([\partial_{x}]=M\), \([u]=M^{2}\) in the unit of mass dimension \(M\). Further, we notice that \([B_{2n+1},\,L]\) does not contain differential operators but it contains only functions. Then we have
+
+\[\frac{\partial u}{\partial t_{2n+1}}=\partial_{x}^{2n+1}u+{\cal O}(u^{2}). \tag{2.6}\]
+
+As Eq.(2.6) is the Lie group type differential equation, we take the Lie algebraic limit. Putting \(u=\epsilon\hat{u}\) first, Eq.(2.6) takes in the form
+
+\[\epsilon\frac{\partial\hat{u}}{\partial t_{2n+1}}=\epsilon\partial_{x}^{2n+1} \hat{u}+{\cal O}(\epsilon^{2}\hat{u}^{2}), \tag{2.7}\]
+
+and afterwards we take the limit \(\epsilon\to 0\), which gives
+
+\[\frac{\partial\hat{u}}{\partial t_{2n+1}}=\partial_{x}^{2n+1}\hat{u}. \tag{2.8}\]
+
+Then for trigonometric/hyperbolic solutions, we see that \(x\) and \(t_{2n+1}\) come in a combination \(X_{i}=a_{i}x+\delta_{i}\to X_{i}=a_{i}x+a_{i}^{2n+1}t_{2n+1}+\delta_{i}\) for 1-soliton solutions. In this way, the time-dependence for trigonometric/hyperbolic solutions is easily determined without knowing details of \(B_{2n+1}\). We can then obtain trigonometric/hyperbolic \(N\)-soliton solutions for the \((2n+1)\)-th order KdV equation from the original KdV \(N\)-soliton solutions just by replacing \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t_{3}+\delta_{i}\to X_{i}^{(2n+1)}=a_{i}x+a_{i}^{2n+1 }t_{2n+1}+\delta_{i},(i=1,2,\cdots,N)\).
+
+For example, the original third order KdV equation is given by 1
+
+Footnote 1: We use the notation \(u_{x}=\partial_{x}u\), \(u_{2x}=\partial_{x}^{2}u\), \(\cdots\), throughout the paper.
+
+\[u_{t_{3}}=u_{3x}-6uu_{x}, \tag{2.9}\]
+
+and the fifth order KdV equation is given by [27],
+
+\[u_{t_{5}}=u_{5x}-10uu_{3x}-20u_{x}u_{2x}+30u^{2}u_{x}. \tag{2.10}\]
+
+These two equations look quite different, but the 1-soliton solution for the third order KdV equation is given by \(z=-a\tanh((ax+a^{3}t+\delta)/2)\), while 1-soliton solution for the fifth order KdV equation is given by \(z=-a\tanh((ax+a^{5}t+\delta)/2)\). In this way, even for any \(N\)-soliton solutions, we can obtain the fifth order KdV solution from third order KdV solution just by replacing \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t+\delta_{i}\to X_{i}^{(5)}=a_{i}x+a_{i}^{5}t+ \delta_{i}\). See more details in the Wazwaz's nice textbook [27].
+
+However, as we explain in the next section, the way to determine the time dependence by taking the Lie algebraic limit does not applicable for elliptic solutions.
+
+## 3 Elliptic solutions for the Lax type higher order KdV equations
+
+We consider here elliptic 1-soliton solutions for higher order KdV equations up to ninth order. We first study whether higher order KdV equations reduces to differential equations of the elliptic curves. If a differential equation of the elliptic curve exists, via dimensional analysis, \([\partial_{x}]=M\), \([u]=M^{2}\), \([k_{3}]=M^{0}\), \([k_{2}]=M^{2}\), \([k_{1}]=M^{4}\), and \([k_{0}]=M^{6}\), that must be the differential equation of the Weierstrass type elliptic curve
+
+\[{u_{x}}^{2}=k_{3}u^{3}+k_{2}u^{2}+k_{1}u+k_{0}, \tag{3.1}\]
+
+where \(k_{i}(i=0,1,2,3)\) are constants. We cannot use the method to take the Lie algebraic limit to find the time dependence of the elliptic 1-soliton solution, because we cannot take \(u\to 0\) as \(k_{0}\neq 0\) is essential in the elliptic case. By differentiating Eq.(3.1), we have the following relations;
+
+\[u_{2x}=\frac{3}{2}k_{3}u^{2}+k_{2}u+\frac{1}{2}k_{1}, \tag{3.2a}\] \[u_{3x}=3k_{3}uu_{x}+k_{2}u_{x},\] (3.2b) \[u_{4x}=3k_{3}uu_{2x}+3k_{3}{u_{x}}^{2}+k_{2}u_{2x},\] (3.2c) \[u_{5x}=9k_{3}u_{x}u_{2x}+3k_{3}uu_{3x}+k_{2}u_{3x},\] (3.2d) \[u_{6x}=12k_{3}u_{x}u_{3x}+9k_{3}{u_{2}}^{2}+3k_{3}uu_{4x}+k_{2}u_ {4x},\] (3.2e) \[u_{7x}=30k_{3}u_{2x}u_{3x}+15k_{3}u_{x}u_{4x}+3k_{3}uu_{5x}+k_{2} u_{5x},\] (3.2f) \[u_{8x}=45k_{3}u_{2x}u_{4x}+30k_{3}{u_{3x}}^{2}+18k_{3}u_{x}u_{5x} +3k_{3}uu_{6x}+k_{2}u_{6x}. \tag{3.2g}\]
+
+### Elliptic solution for the third order KdV(original KdV) equation
+
+The third order KdV (original KdV) equation is given by
+
+\[u_{t_{3}}=u_{3x}-6uu_{x}=\left(u_{2x}-3u^{2}\right)_{x}. \tag{3.3}\]
+
+We consider the 1-soliton solution, where \(x\) and \(t\) come in the combination \(X=x+c_{3}t_{3}+\delta\), then we have
+
+\[u_{2x}-3u^{2}-c_{3}u=\frac{k_{1}}{2}, \tag{3.4}\]
+
+where \(k_{1}/2\) is an integration constant. Further multiplying \(u_{x}\) and integrating, we have the following differential equation of the Weierstrass type elliptic curve
+
+\[{u_{x}}^{2}=2u^{3}+k_{2}u^{2}+k_{1}u+k_{0}, \tag{3.5}\]
+
+where \(k_{2}\), \(k_{1}\), and \(k_{0}\) are constants and \(c_{3}\) is determined as \(c_{3}=k_{2}\), which gives the time-dependence of the 1-soliton solution. If we put \(\wp=u/2+k_{2}/12\), we have the standard differential equation of the Weierstrass \(\wp\) function type
+
+\[\wp_{x}^{2}=4\wp^{3}-g_{2}\wp-g_{3}, \tag{3.6}\]
+
+with
+
+\[g_{2} ={k_{2}}^{2}/12-k_{1}/2, \tag{3.7a}\] \[g_{3} =-{k_{2}}^{3}/216+k_{1}k_{2}/24-k_{0}/4. \tag{3.7b}\]
+
+Elliptic 1-soliton solution is given by
+
+\[u(x,t_{3})=u(X^{(3)})=2\wp(X^{(3)})-\frac{k_{2}}{6}, \tag{3.8}\]
+
+with
+
+\[X^{(3)}=x+c_{3}t_{3}+\delta,\quad c_{3}=k_{2}.\]
+
+It should be noted that we must parametrize the differential equation of the Weierstrass type elliptic curve by \(k_{2}\), \(k_{1}\), and \(k_{0}\) instead of \(g_{2}\) and \(g_{3}\), because coefficients \(c_{2n+1}\) in higher order KdV equations, which determine the time dependence, are expressed with \(k_{2}\), \(k_{1}\), and \(k_{0}\). According to the method of our previous paper, if we find various 1-soliton solutions, we can construct \(N\)-soliton solutions [14].
+
+### Elliptic solution for the fifth order KdV equation
+
+The fifth order KdV equation is given by [27],
+
+\[u_{t_{5}}-\left(u_{4x}-10uu_{2x}-5{u_{x}}^{2}+10u^{3}\right)_{x}=0. \tag{3.9}\]
+
+We consider the elliptic solution, where \(x\) and \(t_{5}\) come in the combination of \(X=x+c_{5}t_{5}+\delta\), which gives
+
+\[c_{5}u-\left(u_{4x}-10uu_{2x}-5{u_{x}}^{2}+10u^{3}\right)+C=0, \tag{3.10}\]
+
+where \(C\) is an integration constant. We will show that the above equation reduces to the same differential equation of the Weierstrass type elliptic curve Eq.(3.1). Substituting Eq.(3.1),
+
+[MISSING_PAGE_FAIL:6]
+
+We take the most general solution i.e. I) case, which is the same differential equation of the elliptic curve as that of the third order KdV equation Eq.(3.5) and \(c_{7}\) is determined as \(c_{7}=-2k_{0}-2k_{1}k_{2}+{k_{2}}^{3}\). Elliptic 1-soliton solution is given by
+
+\[u(x,t_{7})=u(X^{(7)})=2\wp(X^{(7)})-\frac{k_{2}}{6}, \tag{3.21}\]
+
+with
+
+\[X^{(7)}=x+c_{7}t_{7}+\delta,\quad c_{7}=-2k_{0}-2k_{1}k_{2}+{k_{2}}^{3}.\]
+
+### Elliptic solution for the ninth order KdV equation
+
+The ninth order KdV equation is given by [28],
+
+\[u_{t_{9}}-\big{(}u_{8x}-18uu_{6x}-54u_{x}u_{5x}-114u_{2x}u_{4x}-69 u_{3x}{}^{2}+126u^{2}u_{4x}+504uu_{x}u_{3x}\] \[+462{u_{x}}^{2}u_{2x}+378uu_{2x}{}^{2}-630u^{2}{u_{x}}^{2}-420u^{3 }u_{2x}+126u^{5}\big{)}_{x}=0. \tag{3.22}\]
+
+Assuming that \(x\) and \(t_{9}\) come in the combination of \(X=x+c_{9}t_{9}+\delta\), we have
+
+\[c_{9}u-\big{(}u_{8x}-18uu_{6x}-54u_{x}u_{5x}-114u_{2x}u_{4x}-69 u_{3x}{}^{2}+126u^{2}u_{4x}+504uu_{x}u_{3x}\] \[+462{u_{x}}^{2}u_{2x}+378uu_{2x}{}^{2}-630u^{2}{u_{x}}^{2}-420u^{3 }u_{2x}+126u^{5}\big{)}+C=0. \tag{3.23}\]
+
+Substituting Eq.(3.1), \(\cdots\), and Eq.(3.2g) into Eq.(3.23) and comparing coefficients of \(u^{5}\), \(u^{4}\), \(u^{3}\), \(u^{2}\), \(u^{1}\), and \(u^{0}\), we have 6 conditions for 6 constants \(k_{3}\), \(k_{2}\), \(k_{1}\), \(k_{0}\), \(c_{9}\), and \(C\) in the following form
+
+\[{\rm i)} (k_{3}-2)(3k_{3}-2)(3k_{3}-1)(5k_{3}-1)=0, \tag{3.24a}\] \[{\rm ii)} {k_{2}}(k_{3}-2)(3k_{3}-2)(3k_{3}-1)=0,\] (3.24b) \[{\rm iii)} {k_{1}}(k_{3}-2)(3k_{3}-2)(9k_{3}-4)+7{k_{2}}^{2}(k_{3}-2)(3k_{3} -2)=0,\] (3.24c) \[{\rm iv)} {3k_{0}}(k_{3}-2)(225{k_{3}}^{2}-252k_{3}+70)\] (3.24d) \[{\rm v)} {c_{9}}=(675{k_{3}}^{2}-1836k_{3}+966){k_{0}}{k_{2}}+(378{k_{3}}^ {2}-1080k_{3}+651){k_{1}}^{2}/2\] (3.24e) \[{\rm vi)} {C}=(297{k_{3}}^{2}-828k_{3}+462){k_{0}}{k_{1}}/2+(63k_{3}-123){ k_{0}}{k_{2}}^{2}\] (3.24f) \[{\rm iii)} {+}(27k_{3}-57){k_{1}}^{2}{k_{2}}/2+{k_{1}}{k_{2}}^{3}/2.\]
+
+Then we obtain 4 solutions
+
+\[{\rm I)} {k_{3}}=2,\quad{k_{2}},{k_{1}},{k_{0}}:{\rm arbitrary},\quad{c_{9 }}=-6{k_{0}}{k_{2}}+3{k_{1}}^{2}/2-3{k_{1}}{k_{2}}^{2}+{k_{2}}^{4},\] (3.25) \[{C}=-3{k_{0}}{k_{1}}+3{k_{0}}{k_{2}}^{2}-3{k_{1}}^{2}{k_{2}}/2+{k _{1}}{k_{2}}^{3}/2,\] \[{\rm II)} {k_{3}}=2/3,\quad{k_{0}}=(66k_{1}k_{2}-85{k_{2}}^{3})/6,\quad{k_ {2}},{k_{1}}:{\rm arbitrary},\] (3.26) \[{c_{9}}=(99{k_{1}}^{2}+594k_{1}{k_{2}}^{2}-1188{k_{2}}^{4})/2, \quad{C}=(423{k_{1}}^{2}{k_{2}}-2376{k_{1}}{k_{2}}^{3}+with
+
+\[X^{(9)}=x+c_{9}t_{9}+\delta,\quad c_{9}=-6k_{0}k_{2}+3{k_{1}}^{2}/2-3k_{1}{k_{2}}^ {2}+{k_{2}}^{4}.\]
+
+In this way, even for higher order KdV equations, the main structure of the elliptic solution, which is expressed by \(X^{(2n+1)}\), takes the same functional form except the time dependence, that is, \(c_{2n+1}\) in \(X^{(2n+1)}=x+c_{2n+1}t_{2n+1}+\delta\). Compared with the trigonometric/hyperbolic case, \(c_{2n+1}\) becomes complicated for elliptic solutions of higher order KdV equations.
+
+In the general \((2n+1)\)-th order KdV equation, by dimensional analysis \([u_{2nx}]=[u^{n+1}]=M^{2n+2}\), integrated differential equation gives the \((n+1)\)-th order polynomial of \(u\). Then the number of the conditions is \(n+2\), while the number of constants is 6. So, \(n\geq 5\) becomes the overdetermined case, but we expect the existence of the differential equation of the elliptic curve for more than eleventh order KdV equation owing to the nice SO(2,1) Lie group symmetry. Although the existence of such elliptic curve is a priori not guaranteed, we will show later that the elliptic solutions really exist for all higher order KdV equations.
+
+## 4 Backlund transformation for the differential equation of the elliptic curve
+
+Here we will show that the Backlund transformation connects one solution to another solution of the same differential equation of the Weierstrass type elliptic curve. The Lie group structure of KdV equation is given by GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) and the Backlund transformation can be considered as the self gauge transformation of this Lie group. We consider two elliptic solutions for the KdV equation, that is, two solutions \(u^{\prime}(x,t_{3})\) and \(u(x,t_{3})\) for \(u^{\prime}_{t_{3}}-u^{\prime}_{xxx}+6u^{\prime}u^{\prime}_{x}=0\) and \(u_{t_{3}}-u_{xxx}+6uu_{x}=0\). We put the time dependence in the forms; \(X^{\prime}=x+c^{\prime}_{3}t_{3}+\delta^{\prime}\) for \(u^{\prime}(x,t_{3})\) and that of \(X=x+c_{3}t_{3}+\delta\) for \(u(x,t_{3})\). In order to connect two solutions by the Backlund transformation and to construct \(N\)-soliton solutions, \(c^{\prime}_{3}\) and \(c_{3}\) must take the same common value. By integrating twice, we have the same differential equation of the elliptic curve
+
+\[{u^{\prime}_{x}}^{2} =2u^{\prime 3}+k_{2}u^{\prime 2}+k_{1}u^{\prime}+k_{0}, \tag{4.1}\] \[{u_{x}}^{2} =2u^{3}\,+k_{2}u^{2}\,+k_{1}u\,+k_{0}, \tag{4.2}\]
+
+with same coefficients \(k_{2}\), \(k_{1}\), and \(k_{0}\), where we take \(c_{3}=c^{\prime}_{3}=k_{2}\). By taking a constant shift of \(u\to u-k_{2}/6\), we consider the same two differential equations of the Weierstrass type elliptic curve
+
+\[{u^{\prime}_{x}}^{2} =2u^{\prime 3}-2g_{2}u^{\prime}-4g_{3}, \tag{4.3}\] \[{u_{x}}^{2} =2u^{3}\,-2g_{2}u\,-4g_{3}, \tag{4.4}\]
+
+where \(g_{2}\) and \(g_{3}\) are given by Eqs.(3.7a) and (3.7b). It should be mentioned that this differential equation of the Weierstrass type elliptic curve has not only the solution \(u(x)=2\wp(x)\) but also \(N\)-soliton solutions [14].
+
+Here we will show that we can connect two solutions of Eqs.(4.3) and (4.4) by the following Backlund transformation
+
+\[z^{\prime}_{x}+z_{x}=-\frac{a^{2}}{2}+\frac{(z^{\prime}-z)^{2}}{2}, \tag{4.5}\]
+
+where \(u=z_{x}\) and \(u^{\prime}=z^{\prime}_{x}\). We introduce \(U=u^{\prime}+u=z^{\prime}_{x}+z_{x}\) and \(V=z^{\prime}-z\), which gives \(V_{x}=z^{\prime}_{x}-z_{x}=u^{\prime}-u\). Then we have \(u^{\prime}=(U+V_{x})/2\) and \(u=(U-V_{x})/2\). Eqs.(4.3) and (4.4) are given by
+
+\[(U_{x}+V_{xx})^{2} =(U+V_{x})^{3}-4g_{2}(U+V_{x})-16g_{3}, \tag{4.6}\] \[(U_{x}-V_{xx})^{2} =(U-V_{x})^{3}-4g_{2}(U-V_{x})-16g_{3}. \tag{4.7}\]The Backlund transformation (4.5) is given by
+
+\[U=\frac{V^{2}}{2}-\frac{a^{2}}{2}, \tag{4.8}\]
+
+which gives \(U_{x}=VV_{x}\).
+
+First, by taking Eq.(4.6)\(-\)Eq.(4.7), we have
+
+\[U_{x}V_{xx}=\frac{1}{2}\left(3U^{2}V_{x}+{V_{x}}^{3}\right)-2g_{2}V_{x}, \tag{4.9}\]
+
+which reads the form
+
+\[VV_{xx}=\frac{3}{8}\left(V^{2}-a^{2}\right)^{2}+\frac{1}{2}{V_{x}}^{2}-2g_{2}= \frac{1}{2}{V_{x}}^{2}+\frac{3}{8}V^{4}-\frac{3}{4}a^{2}V^{2}+\frac{3}{8}a^{4} -2g_{2}, \tag{4.10}\]
+
+throught the relation (4.8). By dimensional analysis, we have
+
+\[V_{x}^{2}=m_{4}V^{4}+m_{3}V^{3}+m_{2}V^{2}+m_{1}V+m_{0}, \tag{4.11}\]
+
+where \(m_{i}(i=0,1,\cdots,4)\) are constants. By differentiating this relation, we have
+
+\[V_{xx}=2m_{4}V^{3}+\frac{3}{2}m_{3}V^{2}+m_{2}V+\frac{1}{2}m_{1}. \tag{4.12}\]
+
+Substituting this relation into Eq.(4.10), we have
+
+\[2m_{4}V^{4}+\frac{3}{2}m_{3}V^{3}+m_{2}V^{2}+\frac{1}{2}m_{1}V=\frac{1}{2}{V_{ x}}^{2}+\frac{3}{8}V^{4}-\frac{3}{4}a^{2}V^{2}+\frac{3}{8}a^{4}-2g_{2}, \tag{4.13}\]
+
+which gives
+
+\[{V_{x}}^{2} =\left(4m_{4}-\frac{3}{4}\right)V^{4}+3m_{3}V^{3}+\left(2m_{2}+ \frac{3}{2}a^{2}\right)V^{2}+m_{1}V-\frac{3}{4}a^{4}+4g_{2}\] \[=m_{4}V^{4}+m_{3}V^{3}+m_{2}V^{2}+m_{1}V+m_{0}. \tag{4.14}\]
+
+Comparing coefficients of the power of \(V\), we have \(m_{4}=1/4\), \(m_{3}=0\), \(m_{2}=-3a^{2}/2\), \(m_{1}=\) (undetermined), \(m_{0}=-3a^{4}/4+4g_{2}\), which gives
+
+\[{V_{x}}^{2} =\frac{1}{4}V^{4}-\frac{3}{2}a^{2}V^{2}+m_{1}V-\frac{3}{4}a^{4}+4 g_{2}, \tag{4.15}\] \[V_{xx} =\frac{1}{2}V^{3}-\frac{3}{2}a^{2}V+\frac{1}{2}m_{1}. \tag{4.16}\]
+
+Second, by taking Eq.(4.6)\(+\)Eq.(4.7), we have
+
+\[{U_{x}}^{2}+{V_{xx}}^{2}=U^{3}+3{UV_{x}}^{2}-4g_{2}U-16g_{3}. \tag{4.17}\]
+
+Using Eq.(4.8), we have
+
+\[{V^{2}{V_{x}}^{2}+{V_{xx}}^{2}=\left(\frac{V^{2}}{2}-\frac{a^{2}}{2}\right)^{3 }+3\left(\frac{V^{2}}{2}-\frac{a^{2}}{2}\right){V_{x}}^{2}-4g_{2}\left(\frac{V^ {2}}{2}-\frac{a^{2}}{2}\right)-16g_{3}.} \tag{4.18}\]
+
+Substituting \({V_{x}}^{2}\) and \(V_{xx}\) into Eq.(4.18) and by using Eq.(4.15) and Eq.(4.16), we have the condition \({m_{1}}^{2}=4a^{6}-16a^{2}g_{2}-64g_{3}\). Then the undetermined coefficient \(m_{1}\) is determined, and we have the differential equation of the Jacobi type elliptic curve for \(V=z^{\prime}-z\)
+
+\[{V_{x}}^{2}=\frac{1}{4}V^{4}-\frac{3}{2}a^{2}V^{2}{\pm}{\sqrt{4a^{6}-16a^{2}g_ {2}-64g_{3}}}\,V-\frac{3}{4}a^{4}+4g_{2}. \tag{4.19}\]
+
+[MISSING_PAGE_EMPTY:70]
+
+The solution of the Jacobi's inversion problem is that the symmetric combination of \(\mu_{1}(x)\) and \(\mu_{2}(x)\), that is, \(\mu_{1}(x)+\mu_{2}(x)(=u(x)/2)\) and \(\mu_{1}(x)\mu_{2}(x)\) are given by the ratio of the genus two hyperelliptic theta function. However, the above Jacobi's inversion problem is special as the right-hand side of Eq.(5.7) is zero. Then the genus two hyperelliptic theta function takes in the following special 1-variable form \(\vartheta(\pm 2x+d_{1},d_{2})\) where \(d_{1},d_{2}\) are constants, that is, the second argument becomes constant. Then the ratio of such special genus two hyperelliptic theta function is the function of 1-variable \(x\), which becomes proportional to the 1-variable function \(u(x)=2(\mu_{1}(x)+\mu_{2}(x))\). The general genus two hyperelliptic theta function is given by
+
+\[\vartheta(u,v;\tau_{1},\tau_{2},\tau_{12})=\sum_{m,n\in\mathbb{Z}}\exp{\Big{[} i\pi(\tau_{1}m^{2}+\tau_{2}n^{2}+2\tau_{12}mn)+2i\pi(mu+nv)\Big{]}}. \tag{5.9}\]
+
+Then \(F(x,t)=\vartheta(x,d_{2};t,\tau_{2},\tau_{12})\) satisfies the diffusion equation \(\partial_{t}F(x,t)=-i\partial_{x}^{2}F(x,t)/4\pi\). Further, \(F(x,t)\) has the trivial periodicity \(F(x+1,t)=F(x,t)\). It is shown in the Mumford's nice textbook [30] that if \(F(x,t)\) satisfies i) periodicity \(F(x+1,t)=F(x,t)\), ii) diffusion equation \(\partial_{t}F(x,t)=-i\partial_{x}^{2}F(x,t)/4\pi\), \(F(x,t)\) becomes the genus one elliptic theta function of 1-variable \(x\). By solving the Jacobi's inversion problem, the solution \(u(x,t_{5})=u(X^{(5)})=u(x+c_{5}t_{5}+\delta)\) of the fifth order KdV equation is given by the ratio of the special 1-variable hyperelliptic theta function, which gives the elliptic solution. For the \((2n+1)\)-th order KdV equation, the solution of the Jacobi's inversion problem gives \(u(x,t_{2n+1})=u(X^{(2n+1)})\) as the ratio of the special 1-variable genus \(n\) hyperelliptic theta function of the form \(\vartheta(\pm 2x+d_{1},d_{2},\cdots,d_{n})\), which also becomes the genus one elliptic theta function.
+
+For higher order KdV equations, it is shown that solutions are expressed with above special 1-variable hyperelliptic theta functions, which becomes elliptic theta functions. Then we can conclude that all higher order KdV equations always have elliptic solutions, though we have explicitly constructed elliptic solutions only up to the ninth order KdV equation.
+
+## 6 Summary and Discussions
+
+We have studied to construct \(N\)-soliton solution for the Lax type higher order KdV equations by using the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group structure. The main structure of \(N\)-soliton solutions, expressed with \(X_{i}=\alpha_{i}x+\beta_{i}t+\delta_{i},(i=1,2,\cdots,N)\) is the same even for higher order KdV equations. The difference of \(N\)-soliton solutions in various higher order KdV equations is the time dependence, that is, coefficients \(\beta_{i}\).
+
+In trigonometric/hyperbolic solutions, by taking the Lie algebra limit, we can easily determine the time dependence. For the \((2n+1)\)-th order KdV equation, we can obtain \(N\)-soliton solutions from those of the original KdV equation by just the replacement \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t_{3}+\delta_{i}\)\(\to\)\(X_{i}^{(2n+1)}=a_{i}x+a_{i}^{2n+1}t_{2n+1}+\delta_{i}\), \((i=1,2,\cdots,N)\).
+
+For elliptic solutions, up to the ninth order KdV equation, we have obtained \(N\)-soliton solutions from those of the original KdV equation by just the replacement \(X^{(3)}{}_{i}=x+c_{3}t_{3}+\delta_{i}\)\(\to\)\(X^{(2n+1)}{}_{i}=x+c_{2n+1}t_{2n+1}+\delta_{i}\), \((i=1,2,3,4)\) where \(c_{2n+1}\) are given by \(c_{3}=k_{2}\), \(c_{5}=-k_{1}+k_{2}{}^{2}\), \(c_{7}=-2k_{0}-2k_{1}k_{2}+k_{2}{}^{3}\), and \(c_{9}=-6k_{0}k_{2}+3k_{1}{}^{2}/2-3k_{1}k_{2}{}^{2}+k_{2}{}^{4}\) by using coefficients of differential equation of the Weierstrass type elliptic curve \(u_{x}{}^{2}=2u^{3}+k_{2}u^{2}+k_{1}u+k_{0}\).
+
+For general higher order KdV equations, special 1-variable hyperelliptic solutions are known but elliptic solutions are not known so far. Since the same GL(2,\(\mathbb{R}\))\(\cong\) SO(2,1) Lie group structure and the same Backlund transformation exists even for higher order KdV equations, the existence of elliptic solutions will be guaranteed. We can show that elliptic solutions for all higher order KdV equations really exist by the following arguments: For the general \((2n+1)\)-th order KdV equation, it can be formulated in the Jacobi's inversion problem [19, 20], and it is known that there exist solutions expressed with the special 1-variablehyperelliptic theta function of the form \(\vartheta(\pm 2x+d_{1},d_{2},\cdots,n)\)[20, 21, 22, 23, 24], which is shown to be the elliptic theta function according to the Mumford's argument [30]. We can say in another way. As the soliton solution \(u(x,t)=u(X)\), (\(X=\alpha x+\beta t_{2n+1}+\delta\)), which is expressed as the ratio of special 1-variable hyperelliptic theta functions, as it has the trivial periodicity \(X\to X+1\), \(u(X)\) must be the trigonometric/hyperbolic or the elliptic function. Then it becomes the elliptic function according to the Mumford's argument. By using these facts, we can conclude that we always have the elliptic solutions for the general higher order KdV equations.
+
+Further, without using the explicit form of the solution expressed with the \(\wp\) function, we have shown that the KdV type Backlund transformation connects one solution to another solution of the same differential equation of the Weierstrass type elliptic curve.
+
+## References
+
+* [1] C.S. Gardner, J.M. Greene, M.D. Kruskal, and R.M. Miura, Phys. Rev. Lett. **19**, 1095 (1967).
+* [2] P.D. Lax, Commun, Pure and Appl. Math. **21**, 467 (1968).
+* [3] V.E. Zakharov and A.B. Shabat, Sov. Phys. JETP **34**, (1972) 62.
+* [4] M.J. Ablowitz, D.J. Kaup, A.C. Newell, and H. Segur, Phys. Rev. Lett. **31**, 125 (1973).
+* [5] H.D. Wahlquist and F.B. Estabrook, Phys. Rev. Lett. **31**, 1386 (1973).
+* [6] M. Wadati, J. Phys. Soc. Jpn. **36**, 1498 (1974).
+* [7] K. Konno and M. Wadati, Prog. Theor. Phys. **53**, 1652 (1975).
+* [8] R. Hirota, Phys. Rev. Lett. **27**, 1192 (1971).
+* [9] R. Hirota, J. Phys. Soc. Jpn. **33**, 1456 (1972).
+* [10] M. Sato, RIMS Kokyuroku (Kyoto University) **439**, 30 (1981).
+* [11] E. Date, M. Kashiwara, and T. Miwa, Proc. Japan Acad. **57A**, 387 (1981).
+* [12] J. Weiss, J. Math. Phys. **24**, 1405 (1983).
+* [13] M. Hayashi, K. Shigemoto, and T. Tsukioka, Mod. Phys. Lett. **A34**, 1950136 (2019).
+* [14] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **3**, 045004 (2019).
+* [15] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **3**, 085015 (2019).
+* [16] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **4**, 015014 (2020).
+* [17] K. Shigemoto, "The Elliptic Function in Statistical Integrable Models", Tezukayama Academic Review **17**, 15 (2011), [arXiv:1603.01079v2[nlin.SI]].
+* [18] K. Shigemoto, "The Elliptic Function in Statistical Integrable Models II", Tezukayama Academic Review **19**, 1 (2013), [arXiv:1302.6712v1[math-ph]].
+* [19] J.L. Burchnall and T.W. Chaundy, Proc. London Math. Soc. **21**, 420 (1922).
+* [20] E. Date and S. Tanaka, Progr. Theor. Phys. Supplement **59**, 107 (1976).
+* [21] A.R. Its and V.B. Matveev, Theor. Math. Phys. **23**, 343 (1975).
+* [22] H.P. McKean and P. van Moerbeke, Inversions Math. Phys. **30**, 217 (1975).
+* [23] B.A. Dubrobin, V.B. Matveev, and S.P. Novikov, Russian Math. Surveys **31**, 59 (1976).
+* [24] I.M. Krichever, Russian Math. Surveys **32**, 185 (1977).
+
+* [25] I.M. Gel'fand and L.A. Dikii, Funct. Anal. Appl. **12**, 259 (1978)(English).
+* [26] L.A. Dickey, _Soliton equations and Hamiltonian systems_, (World Scientific, Singapore, 2003).
+* [27] A.-M. Wazwaz, _Partial Differential Equations and Solitary Waves Theory_, (Springer-Verlag, Berlin Heidelberg, 2009).
+* [28] Y.-J. Shen, Y.-T. Gao, G.-Q. Meng, Y. Qin and X. Yu, Applied Mathematics and Computation, **274**, 403 (2016).
+* [29] K. Shigemoto, "Jacobi's Inversion Problem for Genus Two Hyperelliptic Integral", Tezukayama Academic Review **20**, 1 (2014), [arXiv:1603.02508v2[math-ph]].
+* [30] D. Mumford, _Tata Lectures on Theta I_, p.4 (Birkhauser, Boston Basel Stuttgart, 1983).

2003.00005v2.mmd

@@ -0,0 +1,294 @@
+# Elliptic Solutions for Higher Order KdV Equations
+
+ Masahito Hayashi
+
+Osaka Institute of Technology, Osaka 535-8585, Japan
+
+Kazuyasu Shigemoto
+
+Tezukayama University, Nara 631-8501, Japan
+
+Takuya Tsukioka
+
+Bukkyo University, Kyoto 603-8301, Japan
+
+masahito.hayashi@oit.ac.jpshigemot@tezukayama-u.ac.jptsukioka@bukkyo-u.ac.jp
+
+###### Abstract
+
+We study higher order KdV equations from the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group point of view. We find elliptic solutions of higher order KdV equations up to the ninth order. We argue that the main structure of the trigonometric/hyperbolic/elliptic \(N\)-soliton solutions for higher order KdV equations is the same as that of the original KdV equation. Pointing out that the difference is only the time dependence, we find \(N\)-soliton solutions of higher order KdV equations can be constructed from those of the original KdV equation by properly replacing the time-dependence. We discuss that there always exist elliptic solutions for all higher order KdV equations.
+
+## 1 Introduction
+
+The soliton system is taken an interest in for a long time by considering that the soliton equation is the concrete example of the exactly solvable nonlinear differential equation [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. Nonlinear differential equation relates to the interesting non-perturbative phenomena, so that studies of the soliton system are important to unveil mechanisms of various interesting physical phenomena such as those in superstring theories. It is quite surprising that such nonlinear soliton equations can be exactly solvable and have \(N\)-soliton solutions. Then we have a dogma that there must be the Lie group structure behind the soliton system, which is a key stone to make nonlinear differential equations exactly solvable.
+
+For the KdV soliton system, the Lie group structure is implicitly built in the Lax operator \(L=\partial_{x}^{2}-u(x,t)\). In order to see the Lie group structure, it is appropriate to formulate by using the linear differential operator \(\partial_{x}\) as the Schrodinger representation of the Lie algebra, which naturally comes to use the AKNS formalism [4] for the Lax equation
+
+\[\frac{\partial}{\partial x}\left(\begin{array}{c}\psi_{1}(x,t)\\ \psi_{2}(x,t)\end{array}\right)=\left(\begin{array}{cc}a/2&-u(x,t)\\ -1&-a/2\end{array}\right)\left(\begin{array}{c}\psi_{1}(x,t)\\ \psi_{2}(x,t)\end{array}\right).\]
+
+Then the Lie group becomes GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) for the KdV equation. An addition formula for elements of this Lie group is the well-known KdV type Backlund transformation.
+
+In our previous papers [13, 14, 15, 16], we have studied GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group approach for the unified soliton systems of KdV/mKdV/sinh-Gordon equations. Using the well-know KdV type Backlund transformation as the addition formula, we have algebraically constructed \(N\)-soliton solutions from various trigonometric/hyperbolic 1-soliton solutions [13, 15, 16]. Since the Lie group structure of KdV equation is the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1), which has elliptic solution, we expect that elliptic \(N\)-soliton solutions for the KdV equation can be constructed by using the Backlund transformation as the addition formula. We then really have succeeded in constructing elliptic \(N\)-soliton solutions [14].
+
+We can interpret this fact in the following way: The KdV equation, which is a typical 2-dimensional soliton equation, has the SO(2,1) Lie group structure and the well-known KdV type Backlund transformation can be interpreted as the addition formula of this Lie group. Then the elliptic function appears as a representation of the Backlund transformation. While, 2-dimensional Ising model, which is a typical 2-dimensional statistical integrable model, has the SO(3) Lie group structure and the Yang-Baxter relation can be interpreted as the addition formula of this Lie group. Then the elliptic function appears as a representation of the Yang-Baxter relation, which is equivalent to the addition formula of the spherical trigonometry [17, 18]. In 2-dimensional integrable, soliton, and statistical models, there is the SO(2,1)/SO(3) Lie group structure behind the model. As representations of the addition formula, the Backlund transformation, and the Yang-Baxter relation, there appears an algebraic function such as the trigonometric/hyperbolic/elliptic functions, which is the key stone to make the 2-dimensional integrable model into the exactly solvable model.
+
+In this paper, we consider Lax type higher order KdV equations and study trigonometric/hyperbolic/elliptic solutions. So far special hyperelliptic solutions for more than the fifth order KdV equation have been vigorously studied by formulating it into the Jacobi's inversion problem [30, 31, 32, 33, 34, 35]. Since the Lie group structure GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) and the Backlund transformation are common even for higher order KdV equations, we expect that there always exist elliptic solutions even for higher order. Then we study to find elliptic solutions up to the ninth order KdV equation, instead of special hyperelliptic solutions. We would like to conclude that we always have elliptic solutions for all higher order KdV equations.
+
+As the application of the third order KdV equation, this equation is first obtained in the analysis of shallow water solitary wave [19]. Even recently, the third order KdV equation becomes important in the analysis of various non-linear phenomena. For example, in the recent interesting works, the third order KdV equation comes out in the analysis of the non-linear acoustic solitary wave in the electron-ion plasma [20, 21, 22, 23]. As the application of the higher order KdV equation, some special fifth order KdV equation(KdV5), which is different from the Lax type equation, is recently experimentally and theoretically interested in. This KdV5 equation comes out in the analysis of various non-linear phenomena, such as cold collisionless plasma [24], gravity-capillary wave [25], shallow water wave with surface tension [26] etc. Theoretically, it is shown that Camassa-Holm equation is transformed into this KdV5 equation [27, 28] and multi-soliton solutions is obtained [29]. In this way, the KdV equation becomes important in the analysis of various non-linear phenomena.
+
+The paper is organized as follows: In section 2, we study trigonometric/hyperbolic solutions for higher order KdV equations. We construct elliptic solutions for higher order KdV equations in section 3. In section 4, we consider the KdV type Backlund transformation as an addition formula for solutions of the Weierstrass type elliptic differential equation. In section 5, we study special 1-variable hyperelliptic solutions, and we discuss a relation between such special 1-variable hyperelliptic solutions and our elliptic solutions. We devote a final section to summarize this paper and to give discussions.
+
+Trigonometric/hyperbolic solutions for the Lax type higher order KdV equations
+
+Lax pair equations for higher order KdV equations are given by
+
+\[L\psi=\frac{a^{2}}{4}\psi, \tag{2.1}\] \[\frac{\partial\psi}{\partial t_{2n+1}}=B_{2n+1}\psi, \tag{2.2}\]
+
+where \(L=\partial_{x}^{2}-u\). By using the pseudo-differential operator \(\partial_{x}^{-1}\), \(B_{2n+1}\) are constructed from \(L\) in the form [36, 37]
+
+\[B_{2n+1}=\left(\mathcal{L}^{2n+1}\right)_{\geq 0}=\partial_{x}^{2n+1}-\frac{2n+ 1}{2}u\partial_{x}^{2n-1}+\cdots, \tag{2.3}\]
+
+with
+
+\[\mathcal{L}=L^{1/2}=\partial_{x}-\frac{u}{2}\partial_{x}^{-1}+\frac{u_{x}}{4} \partial_{x}^{-2}+\cdots,\]
+
+where we denote "\(\geq 0\)" to take positive differential operator parts or function parts for general pseudo-differential operators. The integrability condition gives higher order KdV equations
+
+\[\frac{\partial L}{\partial t_{2n+1}}=[B_{2n+1},\,L]. \tag{2.4}\]
+
+As these higher order KdV equations comes from the Lax formalism, these higher order KdV equations are called the Lax type. There are various higher order KdV equations such as the Sawada-Kotera type, which is the higher order generalization of the Hirota form KdV equation [38]. As operators \(B_{2n+1}\) are constructed from \(L\), higher order KdV equations also have the same Lie group structure GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) as that of the original KdV(=third order KdV) equation. Using \(u=z_{x}\), the KdV type Backlund transformation is given in the form
+
+\[z_{x}^{\prime}+z_{x}=-\frac{a^{2}}{2}+\frac{(z^{\prime}-z)^{2}}{2}, \tag{2.5}\]
+
+which comes from Eq.(2.1) only, so that it is valid even for the higher order KdV equations. In the Lie group approach to the soliton system, if we find 1-soliton solutions, we can construct \(N\)-soliton solutions from various 1-soliton solutions by the Backlund transformation Eq.(2.5) as an addition formula of the Lie group.
+
+For 1-soliton solution of Eq.(2.4), if \(x\) and \(t_{2n+1}\) come in the combination \(X^{(2n+1)}=\alpha x+\beta t_{2n+1}^{\gamma}+\delta\), then if \(\gamma\neq 1\), the right-hand side of Eq.(2.4) is a function of only \(X\), while the left-hand side is a function of \(X\) and \(t\). Therefore, \(\gamma=1\) is necessary, that is, \(X=\alpha x+\beta t_{2n+1}+\delta\). \(N\)-soliton solutions are constructed from various 1-soliton solutions by the Backlund transformation. Then the main structure of \(N\)-soliton solutions, which are expressed with \(X_{i}^{(2n+1)},(i=1,2,\cdots,N)\), takes the same functional forms in higher order KdV equations and in the original KdV equation. The difference is only the time dependence of \(X_{i}=\alpha_{i}x+\beta_{i}t_{2n+1}+\delta_{i},(i=1,2,\cdots,N)\), that is, coefficients \(\beta_{i}\). This is valid not only for the trigonometric/hyperbolic \(N\)-soliton solutions but also for elliptic \(N\)-soliton solutions.
+
+For the trigonometric/hyperbolic \(N\)-soliton solutions, we can easily determine the time dependence without knowing details of \(B_{2n+1}\). For dimensional analysis, we have \([\partial_{x}]=M\), \([u]=M^{2}\) in the unit of mass dimension \(M\). Further, we notice that \([B_{2n+1},\,L]\) does not contain differential operators but it contains only functions. Then we have
+
+\[\frac{\partial u}{\partial t_{2n+1}}=\partial_{x}^{2n+1}u+\mathcal{O}(u^{2}). \tag{2.6}\]As Eq.(2.6) is the Lie group type differential equation, we take the Lie algebraic limit. Putting \(u=\epsilon\hat{u}\) first, Eq.(2.6) takes in the form
+
+\[\epsilon\frac{\partial\hat{u}}{\partial t_{2n+1}}=\epsilon\partial_{x}^{2n+1} \hat{u}+\mathcal{O}(\epsilon^{2}\hat{u}^{2}), \tag{2.7}\]
+
+and afterwards we take the limit \(\epsilon\to 0\), which gives
+
+\[\frac{\partial\hat{u}}{\partial t_{2n+1}}=\partial_{x}^{2n+1}\hat{u}. \tag{2.8}\]
+
+Then for trigonometric/hyperbolic solutions, we see that \(x\) and \(t_{2n+1}\) come in a combination \(X_{i}=a_{i}x+\delta_{i}\to X_{i}=a_{i}x+a_{i}^{2n+1}t_{2n+1}+\delta_{i}\) for 1-soliton solutions. In this way, the time-dependence for trigonometric/hyperbolic solutions is easily determined without knowing details of \(B_{2n+1}\). We can then obtain trigonometric/hyperbolic \(N\)-soliton solutions for the \((2n+1)\)-th order KdV equation from the original KdV \(N\)-soliton solutions just by replacing \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t_{3}+\delta_{i}\to X_{i}^{(2n+1)}=a_{i}x+a_{i}^{2 n+1}t_{2n+1}+\delta_{i},(i=1,2,\cdots,N)\).
+
+For example, the original third order KdV equation is given by 1
+
+Footnote 1: We use the notation \(u_{x}=\partial_{x}u\), \(u_{2x}=\partial_{x}^{2}u\), \(\cdots\), throughout the paper.
+
+\[u_{t_{3}}=u_{3x}-6uu_{x}, \tag{2.9}\]
+
+and the fifth order KdV equation is given by [38],
+
+\[u_{t_{5}}=u_{5x}-10uu_{3x}-20u_{x}u_{2x}+30u^{2}u_{x}. \tag{2.10}\]
+
+These two equations look quite different, but the 1-soliton solution for the third order KdV equation is given by \(z=-a\tanh((ax+a^{3}t+\delta)/2)\), while 1-soliton solution for the fifth order KdV equation is given by \(z=-a\tanh((ax+a^{5}t+\delta)/2)\). In this way, even for any \(N\)-soliton solutions, we can obtain the fifth order KdV solution from third order KdV solution just by replacing \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t+\delta_{i}\to X_{i}^{(5)}=a_{i}x+a_{i}^{5}t+ \delta_{i}\). See more details in the Wazwaz's nice textbook [38].
+
+However, as we explain in the next section, the way to determine the time dependence by taking the Lie algebraic limit does not applicable for elliptic solutions.
+
+## 3 Elliptic solutions for the Lax type higher order KdV equations
+
+We consider here elliptic 1-soliton solutions for higher order KdV equations up to ninth order. We first study whether higher order KdV equations reduces to differential equations of the elliptic curves. If a differential equation of the elliptic curve exists, via dimensional analysis, \([\partial_{x}]=M\), \([u]=M^{2}\), \([k_{3}]=M^{0}\), \([k_{2}]=M^{2}\), \([k_{1}]=M^{4}\), and \([k_{0}]=M^{6}\), that must be the differential equation of the Weierstrass type elliptic curve
+
+\[{u_{x}}^{2}=k_{3}u^{3}+k_{2}u^{2}+k_{1}u+k_{0}, \tag{3.1}\]
+
+where \(k_{i}(i=0,1,2,3)\) are constants. We cannot use the method to take the Lie algebraic limit to find the time dependence of the elliptic 1-soliton solution, because we cannot take \(u\to 0\) as \(k_{0}\neq 0\) is essential in the elliptic case. By differentiating Eq.(3.1), we have the following 
+
+[MISSING_PAGE_FAIL:5]
+
+### Elliptic solution for the fifth order KdV equation
+
+The fifth order KdV equation is given by [38],
+
+\[u_{t_{5}}-\big{(}u_{4x}-10uu_{2x}-5{u_{x}}^{2}+10u^{3}\big{)}_{x}=0. \tag{3.9}\]
+
+We consider the elliptic solution, where \(x\) and \(t_{5}\) come in the combination of \(X=x+c_{5}t_{5}+\delta\), which gives
+
+\[c_{5}u-\big{(}u_{4x}-10uu_{2x}-5{u_{x}}^{2}+10u^{3}\big{)}+C=0, \tag{3.10}\]
+
+where \(C\) is an integration constant. We will show that the above equation reduces to the same differential equation of the Weierstrass type elliptic curve Eq.(3.1). Substituting Eq.(3.1), \(\cdots\), and Eq.(3.2c) into Eq.(3.10) and comparing coefficients of \(u^{3}\), \(u^{2}\), \(u^{1}\), and \(u^{0}\), we have 4 conditions for 6 constants \(k_{3}\), \(k_{2}\), \(k_{1}\), \(k_{0}\), \(c_{5}\), and \(C\) in the form
+
+\[{\rm i)} (k_{3}-2)(3k_{3}-2)=0, \tag{3.11a}\] \[{\rm ii)} k_{2}(k_{3}-2)=0,\] (3.11b) \[{\rm iii)} c_{5}=(9k_{3}/2-10)k_{1}+{k_{2}}^{2},\] (3.11c) \[{\rm iv)} C=(3k_{3}-5)k_{0}+k_{1}k_{2}/2. \tag{3.11d}\]
+
+Then we have two solutions
+
+\[{\rm I)} \quad k_{3}=2,\quad k_{2},k_{1},k_{0}:{\rm arbitrary},\quad c_{5 }=-k_{1}+{k_{2}}^{2},\quad C=k_{0}+k_{1}k_{2}/2, \tag{3.12}\] \[{\rm II)} \quad k_{3}=\frac{2}{3},\quad k_{2}=0,\quad k_{1},k_{0}:{\rm arbitrary },\quad c_{5}=-7k_{1},\quad C=-3k_{0}. \tag{3.13}\]
+
+We here take the most general solution, i.e. I) case, which gives the same differential equation of the elliptic curve \({u_{x}}^{2}=2u^{3}+k_{2}u^{2}+{k_{2}}^{2}u+k_{0}\) as that of the third order KdV equation Eq.(3.5) and \(c_{5}\) is determined as \(c_{5}=-k_{1}+{k_{2}}^{2}\). Elliptic 1-soliton solution is given by
+
+\[u(x,t_{5})=u(X^{(5)})=2\wp(X^{(5)})-\frac{k_{2}}{6}, \tag{3.14}\]
+
+with
+
+\[X^{(5)}=x+c_{5}t_{5}+\delta,\quad c_{5}=-k_{1}+{k_{2}}^{2}.\]
+
+Figure 1: The third order KdV solution. The red line shows \(u(x,0)\) and the blue line shows \(u(x,1)\) with \(k_{0}=1.2\), \(k_{1}=-1.6\), \(k_{2}=0.8\), \(\delta=0\). We can see the time-dependence.
+
+We sketch the graphs of the fifth order KdV solution in Figure 2. The red line shows \(u(x,0)\) and the blue line shows \(u(x,1)\). We take \(k_{0}=1.2\), \(k_{1}=-1.6\), \(k_{2}=0.8\), \(\delta=0\) in the graph. From this graph, we can see difference of the time-dependence of the solution between the third order solution and the fifth order solution.
+
+### Elliptic solution for the seventh order KdV equation
+
+The seventh order KdV equation is given by [38],
+
+\[u_{t_{7}}-\left(u_{6x}-14uu_{4x}-28u_{x}u_{3x}-21{u_{2x}}^{2}+70u^{2}u_{2x}+70 uu_{x}{}^{2}-35u^{4}\right)_{x}=0. \tag{3.15}\]
+
+In this case, assuming that \(x\) and \(t_{7}\) come in the combination of \(X=x+c_{7}t_{7}+\delta\), we have
+
+\[c_{7}u-\left(u_{6x}-14uu_{4x}-28u_{x}u_{3x}-21{u_{2x}}^{2}+70u^{2}u_{2x}+70uu_{ x}{}^{2}-35u^{4}\right)+C=0. \tag{3.16}\]
+
+Repeatedly substituting Eq.(3.1), \(\cdots\), and Eq.(3.2e) into Eq.(3.16) and comparing coefficients of \(u^{4}\), \(u^{3}\), \(u^{2}\), \(u^{1}\), and \(u^{0}\), we have 5 conditions for 6 constants \(k_{3}\), \(k_{2}\), \(k_{1}\), \(k_{0}\), \(c_{7}\), and \(C\) of the form
+
+\[\mathrm{i)} (k_{3}-2)(3k_{3}-2)(3k_{3}-1)=0, \tag{3.17a}\] \[\mathrm{ii)} \ k_{2}(k_{3}-2)(3k_{3}-2)=0,\] (3.17b) \[\mathrm{iii)} \ k_{1}(k_{3}-2)(6k_{3}-5)+3{k_{2}}^{2}(k_{3}-2)=0,\] (3.17c) \[\mathrm{iv)} \ c_{7}=(45{k_{3}}^{2}-126k_{3}+70)k_{0}+(27k_{3}-56)k_{1}k_{2}+{ k_{2}}^{3},\] (3.17d) \[\mathrm{v)} \ C=(15k_{3}-28)k_{0}k_{2}+(9k_{3}-21){k_{1}}^{2}/4+{k_{1}}{k_{2}} ^{2}/2. \tag{3.17e}\]
+
+Then we get 3 solutions
+
+\[\mathrm{I)} \ k_{3}=2,\quad k_{2},k_{1},k_{0}:\mathrm{arbitrary},\quad c_{7} =-2k_{0}-2k_{1}k_{2}+{k_{2}}^{3}, \tag{3.18}\] \[\quad\quad C=2k_{0}k_{2}-3{k_{1}}^{2}/4+{k_{1}}{k_{2}}^{2}/2,\] \[\mathrm{II)} \ k_{3}=2/3,\quad k_{1}=3{k_{2}}^{2},\quad k_{2},k_{0}:\mathrm{ arbitrary},\quad c_{7}=6k_{0}-113{k_{2}}^{3},\] (3.19) \[\quad\quad C=-18k_{0}k_{2}-129{k_{2}}^{4}/4,\] \[\mathrm{III)} \ k_{3}=1/3,\quad k_{2}=0,\quad k_{1}=0,\quad k_{0}:\mathrm{ arbitrary},\quad c_{7}=33k_{0},\quad C=0. \tag{3.20}\]
+
+Figure 2: The fifth order KdV solution. The red line shows \(u(x,0)\) and the blue line shows \(u(x,1)\) with \(k_{0}=1.2\), \(k_{1}=-1.6\), \(k_{2}=0.8\), \(\delta=0\). We can see the difference of the time-dependence between the third order solution and the fifth order solution.
+
+[MISSING_PAGE_FAIL:8]
+
+with
+
+\[X^{(9)}=x+c_{9}t_{9}+\delta,\quad c_{9}=-6k_{0}k_{2}+3{k_{1}}^{2}/2-3k_{1}{k_{2}}^ {2}+{k_{2}}^{4}.\]
+
+In this way, even for higher order KdV equations, the main structure of the elliptic solution, which is expressed by \(X^{(2n+1)}\), takes the same functional form except the time dependence, that is, \(c_{2n+1}\) in \(X^{(2n+1)}=x+c_{2n+1}t_{2n+1}+\delta\). Compared with the trigonometric/hyperbolic case, \(c_{2n+1}\) becomes complicated for elliptic solutions of higher order KdV equations.
+
+In the general \((2n+1)\)-th order KdV equation, by dimensional analysis \([u_{2nx}]=[u^{n+1}]=M^{2n+2}\), integrated differential equation gives the \((n+1)\)-th order polynomial of \(u\). Then the number of the conditions is \(n+2\), while the number of constants is 6. So, \(n\geq 5\) becomes the overdetermined case, but we expect the existence of the differential equation of the elliptic curve for more than eleventh order KdV equation owing to the nice SO(2,1) Lie group symmetry. Although the existence of such elliptic curve is a priori not guaranteed, we will show later that the elliptic solutions really exist for all higher order KdV equations.
+
+## 4 Backlund transformation for the differential equation of the elliptic curve
+
+Here we will show that the Backlund transformation connects one solution to another solution of the same differential equation of the Weierstrass type elliptic curve. The Lie group structure of KdV equation is given by GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) and the Backlund transformation can be considered as the self gauge transformation of this Lie group. We consider two elliptic solutions for the KdV equation, that is, two solutions \(u^{\prime}(x,t_{3})\) and \(u(x,t_{3})\) for \(u^{\prime}_{t_{3}}-u^{\prime}_{xxx}+6u^{\prime}u^{\prime}_{x}=0\) and \(u_{t_{3}}-u_{xxx}+6uu_{x}=0\). We put the time dependence in the forms; \(X^{\prime}=x+c^{\prime}_{3}t_{3}+\delta^{\prime}\) for \(u^{\prime}(x,t_{3})\) and that of \(X=x+c_{3}t_{3}+\delta\) for \(u(x,t_{3})\). In order to connect two solutions by the Backlund transformation and to construct \(N\)-soliton solutions, \(c^{\prime}_{3}\) and \(c_{3}\) must take the same common value. By integrating twice, we have the same differential equation of the elliptic curve
+
+\[{u^{\prime}_{x}}^{2} = 2{u^{\prime}}^{3}+k_{2}{u^{\prime}}^{2}+k_{1}{u^{\prime}}+k_{0}, \tag{4.1}\] \[{u_{x}}^{2} = 2{u^{3}}\,+k_{2}{u^{2}}\,+k_{1}{u}\,+k_{0}, \tag{4.2}\]
+
+with same coefficients \(k_{2}\), \(k_{1}\), and \(k_{0}\), where we take \(c_{3}=c^{\prime}_{3}=k_{2}\). By taking a constant shift of \(u\to u-k_{2}/6\), we consider the same two differential equations of the Weierstrass type elliptic curve
+
+\[{u^{\prime}_{x}}^{2} = 2{u^{\prime}}^{3}-2g_{2}{u^{\prime}}-4g_{3}, \tag{4.3}\] \[{u_{x}}^{2} = 2{u^{3}}\,-2g_{2}{u}\,-4g_{3}, \tag{4.4}\]
+
+where \(g_{2}\) and \(g_{3}\) are given by Eqs.(3.7a) and (3.7b). It should be mentioned that this differential equation of the Weierstrass type elliptic curve has not only the solution \(u(x)=2\wp(x)\) but also \(N\)-soliton solutions [14].
+
+Here we will show that we can connect two solutions of Eqs.(4.3) and (4.4) by the following Backlund transformation
+
+\[z^{\prime}_{x}+z_{x}=-\frac{a^{2}}{2}+\frac{(z^{\prime}-z)^{2}}{2}, \tag{4.5}\]
+
+where \(u=z_{x}\) and \(u^{\prime}=z^{\prime}_{x}\). We introduce \(U=u^{\prime}+u=z^{\prime}_{x}+z_{x}\) and \(V=z^{\prime}-z\), which gives \(V_{x}=z^{\prime}_{x}-z_{x}=u^{\prime}-u\). Then we have \(u^{\prime}=(U+V_{x})/2\) and \(u=(U-V_{x})/2\). Eqs.(4.3) and (4.4) are given by
+
+\[(U_{x}+V_{xx})^{2} = (U+V_{x})^{3}-4g_{2}(U+V_{x})-16g_{3}, \tag{4.6}\] \[(U_{x}-V_{xx})^{2} = (U-V_{x})^{3}-4g_{2}(U-V_{x})-16g_{3}. \tag{4.7}\]The Backlund transformation (4.5) is given by
+
+\[U=\frac{V^{2}}{2}-\frac{a^{2}}{2}, \tag{4.8}\]
+
+which gives \(U_{x}=VV_{x}\).
+
+First, by taking Eq.(4.6)\(-\)Eq.(4.7), we have
+
+\[U_{x}V_{xx}=\frac{1}{2}\left(3U^{2}V_{x}+{V_{x}}^{3}\right)-2g_{2}V_{x}, \tag{4.9}\]
+
+which reads the form
+
+\[VV_{xx}=\frac{3}{8}\left(V^{2}-a^{2}\right)^{2}+\frac{1}{2}{V_{x}}^{2}-2g_{2}= \frac{1}{2}{V_{x}}^{2}+\frac{3}{8}V^{4}-\frac{3}{4}a^{2}V^{2}+\frac{3}{8}a^{4} -2g_{2}, \tag{4.10}\]
+
+through the relation (4.8). By dimensional analysis, we have
+
+\[V_{x}^{2}=m_{4}V^{4}+m_{3}V^{3}+m_{2}V^{2}+m_{1}V+m_{0}, \tag{4.11}\]
+
+where \(m_{i}(i=0,1,\cdots,4)\) are constants. By differentiating this relation, we have
+
+\[V_{xx}=2m_{4}V^{3}+\frac{3}{2}m_{3}V^{2}+m_{2}V+\frac{1}{2}m_{1}. \tag{4.12}\]
+
+Substituting this relation into Eq.(4.10), we have
+
+\[2m_{4}V^{4}+\frac{3}{2}m_{3}V^{3}+m_{2}V^{2}+\frac{1}{2}m_{1}V=\frac{1}{2}{V_{ x}}^{2}+\frac{3}{8}V^{4}-\frac{3}{4}a^{2}V^{2}+\frac{3}{8}a^{4}-2g_{2}, \tag{4.13}\]
+
+which gives
+
+\[{V_{x}}^{2} =\left(4m_{4}-\frac{3}{4}\right)V^{4}+3m_{3}V^{3}+\left(2m_{2}+ \frac{3}{2}a^{2}\right)V^{2}+m_{1}V-\frac{3}{4}a^{4}+4g_{2}\] \[=m_{4}V^{4}+m_{3}V^{3}+m_{2}V^{2}+m_{1}V+m_{0}. \tag{4.14}\]
+
+Comparing coefficients of the power of \(V\), we have \(m_{4}=1/4\), \(m_{3}=0\), \(m_{2}=-3a^{2}/2\), \(m_{1}=\) (undetermined), \(m_{0}=-3a^{4}/4+4g_{2}\), which gives
+
+\[{V_{x}}^{2} =\frac{1}{4}V^{4}-\frac{3}{2}a^{2}V^{2}+m_{1}V-\frac{3}{4}a^{4}+4 g_{2}, \tag{4.15}\] \[V_{xx} =\frac{1}{2}V^{3}-\frac{3}{2}a^{2}V+\frac{1}{2}m_{1}. \tag{4.16}\]
+
+Second, by taking Eq.(4.6)\(+\)Eq.(4.7), we have
+
+\[{U_{x}}^{2}+{V_{xx}}^{2}=U^{3}+3{U{V_{x}}}^{2}-4g_{2}U-16g_{3}. \tag{4.17}\]
+
+Using Eq.(4.8), we have
+
+\[{V^{2}{V_{x}}}^{2}+{V_{xx}}^{2}=\left(\frac{V^{2}}{2}-\frac{a^{2}}{2}\right)^{ 3}+3\left(\frac{V^{2}}{2}-\frac{a^{2}}{2}\right){V_{x}}^{2}-4g_{2}\left(\frac {V^{2}}{2}-\frac{a^{2}}{2}\right)-16g_{3}. \tag{4.18}\]
+
+Substituting \({V_{x}}^{2}\) and \(V_{xx}\) into Eq.(4.18) and by using Eq.(4.15) and Eq.(4.16), we have the condition \({m_{1}}^{2}=4a^{6}-16a^{2}g_{2}-64g_{3}\). Then the undetermined coefficient \(m_{1}\) is determined, and we have the differential equation of the Jacobi type elliptic curve for \(V=z^{\prime}-z\)
+
+\[{V_{x}}^{2}=\frac{1}{4}V^{4}-\frac{3}{2}a^{2}V^{2}\pm\sqrt{4a^{6}-16a^{2}g_{2} -64g_{3}}\,V-\frac{3}{4}a^{4}+4g_{2}. \tag{4.19}\]
+
+[MISSING_PAGE_EMPTY:84]
+
+The solution of the Jacobi's inversion problem is that the symmetric combination of \(\mu_{1}(x)\) and \(\mu_{2}(x)\), that is, \(\mu_{1}(x)+\mu_{2}(x)(=u(x)/2)\) and \(\mu_{1}(x)\mu_{2}(x)\) are given by the ratio of the genus two hyperelliptic theta function. However, the above Jacobi's inversion problem is special as the right-hand side of Eq.(5.7) is zero. Then the genus two hyperelliptic theta function takes in the following special 1-variable form \(\vartheta(\pm 2x+d_{1},d_{2})\) where \(d_{1},d_{2}\) are constants, that is, the second argument becomes constant. Then the ratio of such special genus two hyperelliptic theta function is the function of 1-variable \(x\), which becomes proportional to the 1-variable function \(u(x)=2(\mu_{1}(x)+\mu_{2}(x))\). The general genus two hyperelliptic theta function is given by
+
+\[\vartheta(u,v;\tau_{1},\tau_{2},\tau_{12})=\sum_{m,n\in\mathbb{Z}}\exp{\Big{[} i\pi(\tau_{1}m^{2}+\tau_{2}n^{2}+2\tau_{12}mn)+2i\pi(mu+nv)\Big{]}}. \tag{5.9}\]
+
+Then \(F(x,t)=\vartheta(x,d_{2};t,\tau_{2},\tau_{12})\) satisfies the diffusion equation \(\partial_{t}F(x,t)=-i\partial_{x}^{2}F(x,t)/4\pi\). Further, \(F(x,t)\) has the trivial periodicity \(F(x+1,t)=F(x,t)\). It is shown in the Mumford's nice textbook [41] that if \(F(x,t)\) satisfies i) periodicity \(F(x+1,t)=F(x,t)\), ii) diffusion equation \(\partial_{t}F(x,t)=-i\partial_{x}^{2}F(x,t)/4\pi\), \(F(x,t)\) becomes the genus one elliptic theta function of 1-variable \(x\). By solving the Jacobi's inversion problem, the solution \(u(x,t_{5})=u(X^{(5)})=u(x+c_{5}t_{5}+\delta)\) of the fifth order KdV equation is given by the ratio of the special 1-variable hyperelliptic theta function, which gives the elliptic solution. For the \((2n+1)\)-th order KdV equation, the solution of the Jacobi's inversion problem gives \(u(x,t_{2n+1})=u(X^{(2n+1)})\) as the ratio of the special 1-variable genus \(n\) hyperelliptic theta function of the form \(\vartheta(\pm 2x+d_{1},d_{2},\cdots,d_{n})\), which also becomes the genus one elliptic theta function.
+
+For higher order KdV equations, it is shown that solutions are expressed with above special 1-variable hyperelliptic theta functions, which becomes elliptic theta functions. Then we can conclude that all higher order KdV equations always have elliptic solutions, though we have explicitly constructed elliptic solutions only up to the ninth order KdV equation.
+
+## 6 Summary and Discussions
+
+We have studied to construct \(N\)-soliton solution for the Lax type higher order KdV equations by using the GL(2,\(\mathbb{R}\)) \(\cong\) SO(2,1) Lie group structure. The main structure of \(N\)-soliton solutions, expressed with \(X_{i}=\alpha_{i}x+\beta_{i}t+\delta_{i},(i=1,2,\cdots,N)\) is the same even for higher order KdV equations. The difference of \(N\)-soliton solutions in various higher order KdV equations is the time dependence, that is, coefficients \(\beta_{i}\).
+
+In trigonometric/hyperbolic solutions, by taking the Lie algebra limit, we can easily determine the time dependence. For the \((2n+1)\)-th order KdV equation, we can obtain \(N\)-soliton solutions from those of the original KdV equation by just the replacement \(X_{i}^{(3)}=a_{i}x+a_{i}^{3}t_{3}+\delta_{i}\)\(\rightarrow\)\(X_{i}^{(2n+1)}=a_{i}x+a_{i}^{2n+1}t_{2n+1}+\delta_{i}\), \((i=1,2,\cdots,N)\).
+
+For elliptic solutions, up to the ninth order KdV equation, we have obtained \(N\)-soliton solutions from those of the original KdV equation by just the replacement \(X^{(3)}{}_{i}=x+c_{3}t_{3}+\delta_{i}\)\(\rightarrow\)\(X^{(2n+1)}{}_{i}=x+c_{2n+1}t_{2n+1}+\delta_{i}\), \((i=1,2,3,4)\) where \(c_{2n+1}\) are given by \(c_{3}=k_{2}\), \(c_{5}=-k_{1}+k_{2}{}^{2}\), \(c_{7}=-2k_{0}-2k_{1}k_{2}+k_{2}{}^{3}\), and \(c_{9}=-6k_{0}k_{2}+3k_{1}{}^{2}/2-3k_{1}k_{2}{}^{2}+k_{2}{\(x+c_{2n+1}t_{2n+1}+\delta\) is the same and difference is only the time dependence \(c_{2n+1}\). Then, as the elliptic solution of the third order KdV equation exist with \(X^{(3)}\) variable, the existence of the elliptic solution of all higher order KdV equation with \(X^{(2n+1)}\) is guaranteed.
+
+Second way is to formulate in the Jacobi's inversion problem. For the general \((2n+1)\)-th order KdV equation, it can be formulated in the Jacobi's inversion problem [30, 31], and it is known that there exist solutions expressed with the special 1-variable hyperelliptic theta function of the form \(\vartheta(\pm 2x+d_{1},d_{2},\cdots,n)\)[31, 32, 33, 34, 35], which is shown to be the elliptic theta function according to the Mumford's argument [41]. We can say in another way. As the soliton solution \(u(x,t)=u(X)\), \((X=\alpha x+\beta t_{2n+1}+\delta)\), which is expressed as the ratio of special 1-variable hyperelliptic theta functions, as it has the trivial periodicity \(X\to X+1\), \(u(X)\) must be the trigonometric/hyperbolic or the elliptic function. Then it becomes the elliptic function according to the Mumford's argument.
+
+By using these two different ways, we can conclude that we always have the elliptic solutions for the general higher order KdV equations.
+
+Further, without using the explicit form of the solution expressed with the \(\wp\) function, we have shown that the KdV type Backlund transformation connects one solution to another solution of the same differential equation of the Weierstrass type elliptic curve.
+
+## References
+
+* [1] C.S. Gardner, J.M. Greene, M.D. Kruskal, and R.M. Miura, Phys. Rev. Lett. **19**, 1095 (1967).
+* [2] P.D. Lax, Commun, Pure and Appl. Math. **21**, 467 (1968).
+* [3] V.E. Zakharov and A.B. Shabat, Sov. Phys. JETP **34**, (1972) 62.
+* [4] M.J. Ablowitz, D.J. Kaup, A.C. Newell, and H. Segur, Phys. Rev. Lett. **31**, 125 (1973).
+* [5] H.D. Wahlquist and F.B. Estabrook, Phys. Rev. Lett. **31**, 1386 (1973).
+* [6] M. Wadati, J. Phys. Soc. Jpn. **36**, 1498 (1974).
+* [7] K. Konno and M. Wadati, Prog. Theor. Phys. **53**, 1652 (1975).
+* [8] R. Hirota, Phys. Rev. Lett. **27**, 1192 (1971).
+* [9] R. Hirota, J. Phys. Soc. Jpn. **33**, 1456 (1972).
+* [10] M. Sato, RIMS Kokyuroku (Kyoto University) **439**, 30 (1981).
+* [11] E. Date, M. Kashiwara, and T. Miwa, Proc. Japan Acad. **57A**, 387 (1981).
+* [12] J. Weiss, J. Math. Phys. **24**, 1405 (1983).
+* [13] M. Hayashi, K. Shigemoto, and T. Tsukioka, Mod. Phys. Lett. **A34**, 1950136 (2019).
+* [14] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **3**, 045004 (2019).
+* [15] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **3**, 085015 (2019).
+* [16] M. Hayashi, K. Shigemoto, and T. Tsukioka, J. Phys. Commun. **4**, 015014 (2020).
+* [17] K. Shigemoto, "The Elliptic Function in Statistical Integrable Models", Tezukayama Academic Review **17**, 15 (2011), [arXiv:1603.01079v2[nlin.SI]].
+* [18] K. Shigemoto, "The Elliptic Function in Statistical Integrable Models II", Tezukayama Academic Review **19**, 1 (2013), [arXiv:1302.6712v1[math-ph]].
+* [19] D.J. Korteweg and G. de Vries, "On the change of form of long waves advancing in a rectangular canal, and on a new type of long stationary waves", Phil. Mag. **39**, 422 (1895).
+
+* [20] A. Saha and P. Chatterjee, Astrophysics and Space Science, **349**, 239 (2014).
+* [21] A. Saha and P. Chatterjee, Astrophysics and Space Science, **353**, 169 (2014).
+* [22] D.P. Chapagai, J. Tamang and A. Saha, Zeitschrift fur Naturforschung, **A 75**, 183 (2020).
+* [23] J. Tamang and A. Saha, Physics of Plasmas, **27**, 012105 (2020).
+* [24] T. Kakutani and H. Ono, J. Phys. Soc. Jpn., **26**, 1305 (1969).
+* [25] J.K. Hunter and J. Scheurle, Physica, **D 32**, 253 (1988).
+* [26] W.-R. Sun, W.-R. Shan, Y. Jiang, P. Wang and B. Tian, Eur. Phys. J., **D 69**, 47 (2015).
+* [27] H.R. Dullin, G.A. Gottwalt and D.D. Holm, Fluid Dyn. Res., **33**, 73 (2003).
+* [28] H.R. Dullin, G.A. Gottwalt and D.D. Holm, Physica **D 190**, 1 (2004).
+* [29] A.M. Wazwaz, Acta Physica Polonica, **A 130**, 679 (2016).
+* [30] J.L. Burchnall and T.W. Chaundy, Proc. London Math. Soc. **21**, 420 (1922).
+* [31] E. Date and S. Tanaka, Progr. Theor. Phys. Supplement **59**, 107 (1976).
+* [32] A.R. Its and V.B. Matveev, Theor. Math. Phys. **23**, 343 (1975).
+* [33] H.P. McKean and P. van Moerbeke, Invenions Math. Phys. **30**, 217 (1975).
+* [34] B.A. Dubrobin, V.B. Matveev, and S.P. Novikov, Russian Math. Surveys **31**, 59 (1976).
+* [35] I.M. Krichever, Russian Math. Surveys **32**, 185 (1977).
+* [36] I.M. Gel'fand and L.A. Dikii, Funct. Anal. Appl. **12**, 259 (1978)(English).
+* [37] L.A. Dickey, _Soliton equations and Hamiltonian systems_, (World Scientific, Singapore, 2003).
+* [38] A.-M. Wazwaz, _Partial Differential Equations and Solitary Waves Theory_, (Springer-Verlag, Berlin Heidelberg, 2009).
+* [39] Y.-J. Shen, Y.-T. Gao, G.-Q. Meng, Y. Qin and X. Yu, Applied Mathematics and Computation, **274**, 403 (2016).
+* [40] K. Shigemoto, "Jacobi's Inversion Problem for Genus Two Hyperelliptic Integral", Tezukayama Academic Review **20**, 1 (2014), [arXiv:1603.02508v2[math-ph]].
+* [41] D. Mumford, _Tata Lectures on Theta I_, p.4 (Birkhauser, Boston Basel Stuttgart, 1983).

2003.00007v1.mmd

@@ -0,0 +1,89 @@
+# Generating EEG features from Acoustic features
+
+ Gautam Krishna
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+Co Tran
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+Mason Carnahan*
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+###### Abstract
+
+In this paper we demonstrate predicting electroencephalography (EEG) features from acoustic features using recurrent neural network (RNN) based regression model and generative adversarial network (GAN). We predict various types of EEG features from acoustic features. We compare our results with the previously studied problem on speech synthesis using EEG and our results demonstrate that EEG features can be generated from acoustic features with lower root mean square error (RMSE), normalized RMSE values compared to generating acoustic features from EEG features (ie: speech synthesis using EEG) when tested using the same data sets.
+
+ electroencephalography (EEG), deep learning
+
+## I Introduction
+
+Electroencephalography (EEG) is a non invasive way of measuring electrical activity of human brain. EEG sensors are placed on the scalp of a subject to obtain the EEG recordings. The references [1, 2, 3] demonstrate that EEG features can be used to perform isolated and continuous speech recognition where EEG signals recorded while subjects were speaking or listening, are translated to text using automatic speech recognition (ASR) models. In [4] authors demonstrated synthesizing speech from invasive electrocorticography (ECoG) signals using deep learning models. Similarly in [2, 5] authors demonstrated synthesizing speech from EEG signals using deep learning models. In [2, 5] authors demonstrated results using different types of EEG feature sets. Speech synthesis and speech recognition using EEG features might help people with speaking disabilities or people who are not able to speak with speech restoration.
+
+In this paper we are interested in investigating whether it is possible to predict EEG features from acoustic features. This problem can be formulated as an inverse problem of EEG based speech synthesis. In EEG based speech synthesis, acoustic features are predicted from EEG features as demonstrated by the work explained in references [2, 5]. Predicting EEG features or signatures from unique acoustic patters might help in better understanding of how human brain process speech perception and production. Recording EEG signals in a laboratory is a time consuming and expensive process which requires the use of specialized EEG sensors and amplifiers, thus having a computer model which can generate EEG features from acoustic features might also help with speeding up the EEG data collection process as it is much easier to record speech or audio signal, especially for the task of collecting EEG data for performing speech recognition experiments.
+
+In [6] authors demonstrated medical time series generation using conditional generative adversarial networks [7] for toy data sets. Other related work include the reference [8] where authors demonstrated generating EEG for motor task using wasserstein generative adversarial networks [9]. Similarly in [10] authors generate synthetic EEG using various generative models for the task of steady state visual evoked potential classification. In [11] authors demonstrated EEG data augmentation for the task of emotion recognition. Our work focuses only on generating EEG features from acoustic features.
+
+We first performed experiments using the model used by authors in [5] and then we tried performing experiments using generative adversarial networks (GAN) [12]. In this work we predict various EEG feature sets introduced by authors in [2] from acoustic features extracted from the speech of the subjects as well as from acoustic features extracted from the utterances that the subjects were listening.
+
+Our results demonstrate that predicting EEG features from acoustic features seem to be easier compared to predicting acoustic features from EEG features as the root mean square error (RMSE) values during test time where much lower for predicting EEG features from acoustic features compared to it's inverse problem when tested using the same data sets. To the best of our knowledge this is the first time predicting EEG features from acoustic features is demonstrated using deep learning models.
+
+## II Regression and GAN model
+
+The regression model we used in this work was very similar to the ones used by the authors in [5]. We used the exact training parameters used by authors in [5] for setting values for batch size, number of training epochs, learning rate etc. In [5] authors used only gated recurrent unit (GRU) [13]layers in their model but in this work we also tried performing experiments using Bi directional GRU layers where a forward GRU and backward GRU layer outputs are concatenated to produce the output of the bi directional GRU layer. The architecture of our regression model is described in Figure 1. The model takes acoustic features or mel-frequency cepstral coefficients (MFCC) of dimension 13 as input and outputs EEG features of a specific dimension at every time step. The dimension of the EEG features outputted depends on the EEG feature set used during training, as each EEG feature set had a different dimension value. The time distributed dense layer in the model has number of hidden units equal to the dimension of the EEG feature set used. The mean squared error (MSE) function was used as the regression loss function for the model. The Figure 4 shows the training convergence for the regression model when Bi directional GRU layers were used. There were two Bi-GRU layers with 256 and 128 hidden units respectively.
+
+Generative adversarial network (GAN) [12] consists of two networks namely the generator model and the discriminator model which are trained simultaneously. The generator model learns to generate data from a latent space and the discriminator model evaluates whether the data generated by the generator is fake or is from true data distribution. The training objective of the generator is to fool the discriminator. The main motivation behind trying to perform experiments using GAN was in the case of GAN the loss function is learned where as in regression a fixed loss function (MSE) is used. However GAN models are extremely difficult to train.
+
+Our generator model, as shown in Figure 2, consists of two layers of Bi-GRU with 256, 128 hidden units respectively in each layer followed by a time distributed dense layer with hidden units equal to the dimension of EEG feature set. During training, real MFCC features with dimension 13 from training set are fed into the generator model and the generator outputs a vector of dimension equal to EEG feature set dimension, which can be considered as fake EEG.
+
+The discriminator model, as described in Figure 3, consists of two single layered Bi-GRU with 256, 128 hidden units connected in parallel. At each training step a pair of inputs are fed into the discriminator. The discriminator takes (real MFCC features, fake EEG) and (real MFCC features, real EEG) pairs. The outputs of the two parallel Bi-GRU's are concatenated and then fed to a GRU layer with 128 hidden units. The last time step of the GRU layer is fed into the dense layer with sigmoid activation function.
+
+In order to define the loss functions for both our generator and discriminator model let us first define few terms. Let \(P_{s_{f}}\) be the sigmoid output of the discriminator for (real MFCC features, fake EEG) input pair and let \(P_{s_{e}}\) be the sigmoid output of the discriminator for (real MFCC features, real EEG) input pair during training time. Then we can define the loss function of generator as \(-\log(P_{s_{f}})+(realEEG-fakeEEG)^{2}*0.5\) and loss function of discriminator as \(-\log(P_{s_{e}})-\log(1-P_{s_{f}})\). The weights of Bi-GRU layers in the generator model were initialized with weights of the regression model for easier training. During test time, the trained generator model takes acoustic features or MFCC from test set as input and produces EEG features as output.
+
+The Figure 6 shows the generator model training loss and Figure 7 shows the discriminator model training loss. The GAN model was trained for 200 epochs using adam optimizer with a batch size of 32.
+
+Fig. 1: Regression Model
+
+Fig. 2: Generator in GAN Model
+
+## III Data Sets used for performing experiments
+
+We used the data set used by authors in [5] for performing experiments. The data set contains the simultaneous speech and EEG recording for four subjects. For each subject we used 80% of the data as the training set, 10% as validation set and remaining 10% as test set. This was the main data set used in this work for comparisons. More details of the data set is covered in [5]. We will refer this data set as data set A in this paper.
+
+We also performed some experiments using data set B used by authors in [2]. For this data set we didn't perform experiments for each subject instead we used 80% of the total data as training set, 10% as validation set and remaining 10% as test set. More details of the data set is covered in [2]. We will refer this data set as data set B in this paper. The train-test split was done randomly.
+
+The EEG data used in these data sets were recorded using wet EEG electrodes. In total 32 EEG sensors were used including one electrode as ground as shown in Figure 5. The Brain Product's ActiChamp EEG amplifier was used in the experiments to collect data.
+
+## IV EEG feature extraction details
+
+We followed the same preprocessing methods used by authors in [1, 2, 3, 5] for preprocessing EEG and speech data.
+
+EEG signals were sampled at 1000Hz and a fourth order IIR band pass filter with cut off frequencies 0.1Hz and 70Hz was applied. A notch filter with cut off frequency 60 Hz was used to remove the power line noise. The EEGlab's [14] Independent component analysis (ICA) toolbox was used to remove biological signal artifacts like electrocardiography (ECG), electromyography (EMG), electrooculography (EOG) etc from the EEG signals. We then extracted the three EEG feature sets explained by authors in [2]. The details of each EEG feature set are covered in [2]. Each EEG feature set was extracted at a sampling frequency of 100 Hz for each EEG channel [3].
+
+The recorded speech signal was sampled at 16KHz frequency. We extracted mel-frequency cepstral coefficients (MFCC) of dimension 13 as features for speech signal. The MFCC features were also sampled at 100Hz same as the sampling frequency of EEG features.
+
+Fig. 4: Bi-GRU training loss convergence
+
+Fig. 5: EEG channel locations for the cap used in our experiments
+
+Fig. 3: Discriminator in GAN Model
+
+Fig. 6: Generator training loss
+
+[MISSING_PAGE_FAIL:4]
+
+[MISSING_PAGE_FAIL:5]

2003.00007v2.mmd

@@ -0,0 +1,77 @@
+# Generating EEG features from Acoustic features
+
+ Gautam Krishna
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+Co Tran
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+Mason Carnahan*
+
+_Brain Machine Interface Lab_
+
+_The University of Texas at Austin_
+
+Austin, Texas
+
+###### Abstract
+
+In this paper we demonstrate predicting electroencephalography (EEG) features from acoustic features using recurrent neural network (RNN) based regression model and generative adversarial network (GAN). We predict various types of EEG features from acoustic features. We compare our results with the previously studied problem on speech synthesis using EEG and our results demonstrate that EEG features can be generated from acoustic features with lower root mean square error (RMSE), normalized RMSE values compared to generating acoustic features from EEG features (ie: speech synthesis using EEG) when tested using the same data sets.
+
+ electroencephalography (EEG), deep learning
+
+## I Introduction
+
+Electroencephalography (EEG) is a non invasive way of measuring electrical activity of human brain. EEG sensors are placed on the scalp of a subject to obtain the EEG recordings. The references [1, 2, 3] demonstrate that EEG features can be used to perform isolated and continuous speech recognition where EEG signals recorded while subjects were speaking or listening, are translated to text using automatic speech recognition (ASR) models. In [4] authors demonstrated synthesizing speech from invasive electrocorticography (ECoG) signals using deep learning models. Similarly in [2, 5] authors demonstrated synthesizing speech from EEG signals using deep learning models. In [2, 5] authors demonstrated results using different types of EEG feature sets. Speech synthesis and speech recognition using EEG features might help people with speaking disabilities or people who are not able to speak with speech restoration.
+
+In this paper we are interested in investigating whether it is possible to predict EEG features from acoustic features. This problem can be formulated as an inverse problem of EEG based speech synthesis. In EEG based speech synthesis, acoustic features are predicted from EEG features as demonstrated by the work explained in references [2, 5]. Predicting EEG features or signatures from unique acoustic patters might help in better understanding of how human brain process speech perception and production. Recording EEG signals in a laboratory is a time consuming and expensive process which requires the use of specialized EEG sensors and amplifiers, thus having a computer model which can generate EEG features from acoustic features might also help with speeding up the EEG data collection process as it is much easier to record speech or audio signal, especially for the task of collecting EEG data for performing speech recognition experiments.
+
+In [6] authors demonstrated medical time series generation using conditional generative adversarial networks [7] for toy data sets. Other related work include the reference [8] where authors demonstrated generating EEG for motor task using wasserstein generative adversarial networks [9]. Similarly in [10] authors generate synthetic EEG using various generative models for the task of steady state visual evoked potential classification. In [11] authors demonstrated EEG data augmentation for the task of emotion recognition. Our work focuses only on generating EEG features from acoustic features.
+
+We first performed experiments using the model used by authors in [5] and then we tried performing experiments using generative adversarial networks (GAN) [12]. In this work we predict various EEG feature sets introduced by authors in [2] from acoustic features extracted from the speech of the subjects as well as from acoustic features extracted from the utterances that the subjects were listening.
+
+Our results demonstrate that predicting EEG features from acoustic features seem to be easier compared to predicting acoustic features from EEG features as the root mean square error (RMSE) values during test time were much lower for predicting EEG features from acoustic features compared to it's inverse problem when tested using the same data sets. To the best of our knowledge this is the first time predicting EEG features from acoustic features is demonstrated using deep learning models.
+
+## II Regression and GAN model
+
+The regression model we used in this work was very similar to the ones used by the authors in [5]. We used the exact training parameters used by authors in [5] for setting values for batch size, number of training epochs, learning rate etc. In [5] authors used only gated recurrent unit (GRU) [13]
+
+[MISSING_PAGE_FAIL:2]
+
+## III Data Sets used for performing experiments
+
+We used the data set used by authors in [5] for performing experiments. The data set contains the simultaneous speech and EEG recording for four subjects. For each subject we used 80% of the data as the training set, 10% as validation set and remaining 10% as test set. This was the main data set used in this work for comparisons. More details of the data set is covered in [5]. We will refer this data set as data set A in this paper.
+
+We also performed some experiments using data set B used by authors in [2]. For this data set we didn't perform experiments for each subject instead we used 80% of the total data as training set, 10% as validation set and remaining 10% as test set. More details of the data set is covered in [2]. We will refer this data set as data set B in this paper. The train-test split was done randomly.
+
+The EEG data used in these data sets were recorded using wet EEG electrodes. In total 32 EEG sensors were used including one electrode as ground as shown in Figure 5. The Brain Product's ActiChamp EEG amplifier was used in the experiments to collect data.
+
+## IV EEG feature extraction details
+
+We followed the same preprocessing methods used by authors in [1, 2, 3, 5] for preprocessing EEG and speech data.
+
+EEG signals were sampled at 1000Hz and a fourth order IIR band pass filter with cut off frequencies 0.1Hz and 70Hz was applied. A notch filter with cut off frequency 60 Hz was used to remove the power line noise. The EEGlab's [14] Independent component analysis (ICA) toolbox was used to remove biological signal artifacts like electrocardiography (ECG), electromyography (EMG), electrooculography (EOG) etc from the EEG signals. We then extracted the three EEG feature sets explained by authors in [2]. The details of each EEG feature set are covered in [2]. Each EEG feature set was extracted at a sampling frequency of 100 Hz for each EEG channel [3].
+
+The recorded speech signal was sampled at 16KHz frequency. We extracted mel-frequency cepstral coefficients (MFCC) of dimension 13 as features for speech signal. The MFCC features were also sampled at 100Hz same as the sampling frequency of EEG features.
+
+Fig. 4: Bi-GRU training loss convergence
+
+Fig. 5: EEG channel locations for the cap used in our experiments
+
+Fig. 3: Discriminator in GAN Model
+
+Fig. 6: Generator training loss
+
+[MISSING_PAGE_FAIL:4]
+
+[MISSING_PAGE_FAIL:5]

2003.00009v1.mmd

@@ -0,0 +1,203 @@
+Cell Mechanics Based Computational Classification of Red Blood Cells Via Machine Intelligence Applied
+
+to Morpho-Rheological Markers
+
+Yan Ge, Philipp Rosendahl, Claudio Duran, Nicole Topfner, Sara Ciucci, Jochen Guck*, and Carlo Vittorio Cannistraci*
+
+Y. Ge, S. Ciucci, C. Duran, and C. V. Cannistraci are with Biomedical Cybernetics Group, Biotechnology Center (BIOTEC), Center for Molecular and Cellular Bioengineering (CMCB), Center for Systems Biology Dresden (CSBD), Department of Physics, Technische Universitat Dresden, Tatzberg 47/49, 01307 Dresden, Germany. (*Corresponding author: kalokanghos.agon@gmail.com)Y.Ge, P. Rosendahl and J. Guck are with Cellular Machines Group, Biotechnology Center, Center of Molecular and Cellular Bioengineering, Technische Universitat Dresden, Dresden, Germany. (*Corresponding author: jochen.guck@tu-dresden.de)N. Topfner is with Department of Pediatrics, University Clinic Carl Gustav Carus, Technische Universitat Dresden, Dresden, Germany.C. V. Cannistraci is with Complex Network Intelligence Center, Tsinghua Laboratory of Brain and Intelligence, Tsinghua University, Beijing, China.
+
+###### Abstract
+
+Despite fluorescent cell-labelling being widely employed in biomedical studies, some of its drawbacks are inevitable, with unsuitable fluorescent probes or probes inducing a functional change being the main limitations. Consequently, the demand for and development of label-free methodologies to classify cells is strong and its impact on precision medicine is relevant. Towards this end, high-throughput techniques for cell mechanical phenotyping have been proposed to get a multidimensional biophysical characterization of single cells. With this motivation, our goal here is to investigate the extent to which an unsupervised machine learning methodology, which is applied exclusively on morpho-rheological markers obtained by real-time deformability and fluorescence cytometry (RT-FDC), can address the difficult task of providing label-free discrimination of reticulocytes from mature red blood cells. We focused on this problem, since the characterization of reticulocytes (their percentage and cellular features) in the blood is vital in multiple human disease conditions, especially bone-marrow disorders such as anemia and leukemia. Our approach reports promising label-free results in the classification of reticulocytes from mature red blood cells, and it represents a step forward in the development of high-throughput morpho-rheological-based methodologies for the computational categorization of single cells. Besides, our methodology can be an alternative but also a complementary method to integrate with existing cell-labelling techniques.
+
+ fluorescence marker, cell mechanics, real-time deformability and fluorescence cytometry, unsupervised machine learning, PC-corr, mature red blood cell, reticulocyte, marker prediction +
+Footnote †: publication: 1545-5963 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See [http://www.ieee.org/publications_standards/publications/rights/index.html](http://www.ieee.org/publications_standards/publications/rights/index.html) for more information.
+
+## I Introduction
+
+In biology, a fluorescent tag is a molecule that is chemically bound to aid in the labelling and detection of a biomolecule, and therefore serves as a label or probe. Despite the great success of fluorescent labelling, some of its shortcomings are inevitable. Some probes/labels are incompatible with live cell analysis, for example, antibody labelling against histone modifications[1], or fluorescent reporters for actin are excluded from specific filament structures during filament assembly, resulting in failed signal detection[2]. Even if live cell reporters are available[3], these may have confounding effects on the cells, such as the case of inducing single-strand DNA breaks[4] or impairing chromatin organization and leading to histone dissociation[5]. Besides, in some cases, the label can affect protein functions, or can be toxic and sometimes interfere with normal biological processes[6]. Therefore, an assay that reduces the number of, or even eliminates fluorescent labels required to identify cell phenotypes, is particularly attractive.
+
+The call for label-free assay coincides with cell mechanical characterization. Cell mechanical properties are very often related to cell state and function, thus they can serve as an intrinsic biophysical marker[7]. As a powerful tool, cell mechanics can be used to characterize cells, to monitor their mechanical behaviour and to diagnose pathological alterations[8]. Real-time deformability and fluorescence cytometry (RT-FDC) is a microfluidic high-throughput method for morpho-rheological characterization of single cells[9]. For each cell, multiple morpho-rheological parameters are recorded in real-time and then analysed on-the-fly or in a post-processing step. In addition, also fluorescence detection and even 1-D fluorescence imaging can be performed, and the information can be correlated with the label-free morpho-rheological characterization.
+
+In this study, we investigated how to predict cell type without fluorescence labelling by using the RT-FDC data on a case study with computational approach. To be more specific, our main aims are two: 1) to investigate the problem of computational classification of mature red blood cells (mRBCs) and reticulocytes (RETs) - derived from human blood - considering only morpho -rheological cell features. 2) to investigate the extent to which a basic unsupervised and linear approach performs (in comparison to supervised approaches) to discriminate mRBCs and reticulocytes (RETs) on the exclusive basis of morpho-rheological phenotype data obtained from RT-FDC. We focused on this classification task because the investigation of RETs (their percentage and cellular features) in the blood is an important indicator to differentiate between multiple human diseases[10]. As reticulocyte count is an important sign of erythropoietic activity, it can help e.g. to evaluate different types of anaemia, which is a deficiency in the number or quality of red blood cells. Whereas in acute bleeding or in hemolysis the reticulocyte count is increased (or stable), a low reticulocyte count can indicate dysplastic or aplastic bone marrow disorders, resulting in an impaired erythropoiesis. In addition to quantitative changes, the RETs can change their mechanical properties and become progressively more deformable as they mature towards their normal state, a characteristic that facilitates their release from the functional healthy bone marrow[11].
+
+Mature human red blood cells are characterized by the lack of a nucleus and consequently the absence of transcriptional activity, so that neither DNA nor RNA is typically present in these cells. In contrast, immature red blood cells can be identified by the presence of remaining amounts of nucleic acid, which can be labeled and detected using intercalating dyes such as Hoechst, DAPI or syto13. Indeed, staining of RNA in reticulocytes is a (gold-)standard procedure in clinical blood counts. Here, Nucleic acid dye, syto 13, is used as a fluorescent probe for the ground-truth label information to evaluate our classification performance. We controlled factors associated with fluorescence label issues in order to generate a bone-fide dataset. These data were obtained with a high level of confidence and low noise because the fluorescence labels were adopted according to standard procedures which ensure the respect of staining ability. In our presented pipeline, we adopted a robust unsupervised machine learning procedure and used the PC-corr[12] algorithm to extract the most discriminative markers and their correlations, which were used subsequently to classify mRBCs and RETs. Since the number of RETs in the blood is much smaller than the number of mRBCs, this classification task represents a challenging benchmark to test the proposed machine learning procedure. In addition, label-free classification of mRBCs and RETs based on cell morpho-rheological markers is a very complicated task, and as far as we know there is not any literature on the application of machine learning to this problem, therefore this represents also an innovative topic to consider for precision medicine. We successfully infer a robust combinatorial-marker (a single composed-marker that is defined as mathematical combination of several morpho-rheological markers) and define an appropriate marker threshold that can offer two-group-classification (mRBCs or RETs) of uncategorized cells with acceptable accuracy. The workflow presented hereafter can be generalized and applied to identify other cellular phenotypes (e.g., healthy vs cancer cell, marker positive vs marker negative cell) starting from multidimensional cell-mechanical measures.
+
+## II Methodology
+
+### _Ethical Statement_
+
+With ethical approval for the study (EK89032013) from the ethics committee of the Technische Universitat Dresden, we obtained blood from healthy donors with their informed consent in accordance with the guidelines of good practice and the Declaration of Helsinki. However, they are regarded as three potential patients for research purpose who will go for blood check in this study. Indeed, any patient who needs a diagnosis can be healthy or pathological.
+
+### _Data collection and generation of training and validation set_
+
+Capillary blood was collected after finger trick from three donors (P1, P2, P3) with a 21G, 1.8 mm safety-lancet (Sarstedt AG & Co.). A volume of 2 ul blood was diluted in 1mL of 0.5% methyl cellulose complemented with 2.5 ul Msqt13 nucleic acid stain (Thermo Fisher Scientific Inc., S7575) and incubated 5 minutes at room temperature. RETs contain some RNA in the cytosol that they completely lose during maturation towards mRBCs, therefore they can be distinguished by RNA content. RNA staining enables measurement of RNA content which is related to the maturity of the red blood cells since they lose RNA gradually over a time of ca. one day[13]. Afterwards, all samples were measured by RT-FDC, which not only detects the mechanical phenotype of each individual cell (normal RT-DC[14, 15], characterized by ten features: area, area ratio, aspect, brightness, brightness standard deviation, deformation, inertia ratio, inertia ratio raw, x-size and y-size; see next section for more details about their descriptions) but also simultaneously gather its fluorescence intensity in a manner similar to flow cytometry. This directly correlates mechanical data with fluorescence data based on nucleic acid staining. There is a natural unbalanced cell-group composition in each donor, i.e., the percentage of RETs is much smaller than mRBCs. Since sample P1 contained more RETs in comparison to P2 and P3, we decided to adopt it for deriving the training set. Therefore, considering P1 donor, which comprises of 15,763 mRBCs and 357 RETs, 10,763 mRBCs and 257 RETs were used to create the training set named P1-partition1. The remaining 5,000 mRBCs and 100 RETs were used to create the independent _internal_ (we use the word _internal_ because the validation is based on cells coming from the same donor used for training) validation set, named P1-partition2. The other two donors, P2 and P3, were taken as independent _external_ (because the cells are derived from donors different from the one adopted for training) validation sets, which contains 16,671 mRBCs & 145 RETs, and 15,511 mRBCs & 103 RETs, respectively. To facilitate replication of the results, these data are available for open access as supplementary data.
+
+### _Descriptions of the ten morpho-rheological features_
+
+RT-DC detects the morpho-rheological properties of each single cell, which goes through the microfluidic channel, and represents them with ten numerical features:
+
+1. area: the cell's cross-sectional area derived from the contour.
+2. area ratio: the ratio between the area of the convex hull of the cell's contour and the area of the cell's contour
+3. x-size: the maximal axial-length of the cell suspended in the flow that pass through the channel along the horizontal dimension
+4. y-size: the maximal axial-length of the cell suspended in the flow that pass through the channel along the vertical dimension
+5. aspect ratio: x-size/y-size
+6. brightness: the average brightness value of the pixels inside cell's contour
+7. brightness standard deviation: standard deviation of the pixels' brightness values inside cell's contour
+8. deformation: the deformation of a cell is defined as D = 1 - c, where c is the circularity of the contour. Circularity is defined as: \(c=\frac{2\sqrt{\pi\;area}}{perimeter}\)
+9. inertia ratio: ratio of the image moments[16] of the convex hull of the contour. This parameter is similar to the aspect ratio but has sub-pixel accuracy.
+10. inertia ratio raw: same as above but for the raw contour (no convex hull applied)
+
+### _Unsupervised dimension reduction machine learning procedure_
+
+We adopted PCA, which is a machine learning method for unsupervised linear and parameter-free dimension reduction. We performed unsupervised analysis instead of a supervised one, because it is less prone to overfitting as shown in previous studies[12, 17, 18]. 10,000 resampled datasets were generated from the original training set (P1-Partition 1), each of which was obtained by randomly selecting 200 mRBCs and 200 RETs. We will refer here and in the remainder of the text to this as class-balance procedure. PCA was used to project the data into the first three dimensions of embedding (the first three principal components). We considered only the first three dimensions of embedding since, in general, they form a reduced 3D space of representation to the original multidimensional data, where the patterns associated with the major data variability are compressed. This procedure was repeated 10,000 times, one for each of the resampled datasets. We created balanced datasets because PCA is a data-driven approach and the unbalanced datasets would impair its performance since learning algorithms often fail to generalize inductive rules over the sample space when presented with this form of imbalance[19]. We stress that the procedure to project the data is based on unsupervised dimensional reduction learning because we never used the labels to learn the multivariate transformation that projects the data onto the low-dimensional space.
+
+Next, we considered the labels of the samples (without performing any learning procedure) just to reveal the extent to which the PC1, PC2 and PC3 are able to discriminate the two sample classes. For this, we used the p-value obtained by Mann-Whitney \(U\) test[20] and AUC-ROC to evaluate the mRBCs vs RETs separation on each single dimension, and then summarized the mean p-value and the mean AUC-ROC by considering the 10,000 resampled datasets (Table I).
+
+### _PC-corr discriminative networks and combinatorial marker design_
+
+PC-corr[12] is an algorithm able to enlighten discriminative network functional modules associated with the most discriminant dimension of PCA, which in our case was PC2. We applied the PC-corr algorithm to each of the 10,000 datasets and we considered the mean discriminative networks (obtained as mean of 10,000 networks) associated with the PC2 separation. We applied a cut-off of 0.6 (see Results section C. for more detail) on the weights of this mean discriminative network to extract the modules of PC2-related-features and we detected a unique discriminative network module composed by three morpho-rheological-features (Fig.2A). Then, the features that are engaged in the module of highest association with the PCA discrimination can be mathematically combined (using their mean) to offer a unique value that is named the combinatorial marker. As clarified in the result section, we considered all the possible combinations of the three morpho-rheological-features in order to design potential combinatorial markers to test in the validations. Hence, we designed four candidate combinatorial markers based on the three scaled (using z-score transformation) individual features: the mean of area, y-size and x-size; the mean of area and y-size; the mean of area and x-size; the mean of y-size and x-size.
+
+### _Validation of the designed combinatorial markers_
+
+The validation set P1-partition2, which is composed of 5,000 mRBCs and 100 RETs, was used to create 10,000 resampled datasets. We randomly selected 100 samples from the 5000 mRBCs and merged them with the unique 100 RETs for each resampling population. We used the p-value obtained by Mann-Whitney U test and AUC-ROC to evaluate the classification performance of the combinatorial markers and the single markers. The mean p-value and mean AUC-ROC were calculated based on the 10,000 resampled datasets (Table II).
+
+We clarify that the learning of the combinatorial feature selection on the P1-partion1 dataset is data-driven and unsupervised by performing the PCA and the discriminative network analysis by PC-corr. However, in the next section we describe how to supervisedly detect the optimal operator point (marker threshold) of this marker for the further class prediction on the validation datasets. Therefore, the word unsupervised in the remainder of the article refers to the way we build the marker and not to the way we select the marker threshold.
+
+### _Marker threshold learning and evaluation_
+
+We used the P1-partition1 as the training set to get the optimal operator point (which is the point on the ROC curve that offers the highest AUC for the classification of the two different categories of cells) for the combinatorial markers obtained as the mean of area and y-size. In this case we used a supervised procedure (which is a hypothesis-driven procedure that exploits the training set labels to learn a threshold) and therefore we had to employ a 10-fold cross-validation (first divide the training set to ten partitions, use the nine partitions to learn the optimal operator point, and test it on the remaining
+
+one partition to get its performance according to AUC-ROC). This 10-fold cross-validation procedure was repeated ten times. Each time the ten partitions were created independently at random starting from the original training set (including 10,763 mRBCs and 257 RETs), but in this case, we preserved the original count ratio of mRBCs and RETs (10,763/257=41.9) in every partition. This means that we did not balance the training dataset by considering the same amount of mRBCs and RETs, because the procedure of learning the marker threshold is supervised and hypothesis-driven and we wanted that the optimal operator point (the marker threshold used to decide whether a cell belongs to mRBC or RET) could be learned considering the natural cell unbalance occurring in the blood samples. As a second option, we implemented the same cross-validation procedure above, but for each step we applied a class-balance procedure, such as we did for the unsupervised dimension reduction projection with mRBC resampling. This means that each cross-validation fold was composed of 257 mRBCs (sampled uniformly at random from the 10,763) and 257 RETs. Unfortunately, class-balance learning offered poor results (data not shown). This bad performance of the class-balance procedure is motivated by the fact that here the learning of a pure threshold for a marker value, and not a model to create the marker itself, is implemented. Therefore, the original cell-ratio offers advantages to learn the marker threshold value.
+
+We obtained an array with 100 values, with each element specifying the optimal operator point generated by the unbalanced composition of mRBCs and RETs. More precisely, we firstly z-score-scaled each dataset and then classified each cell by comparing the learned threshold with its mean of z-score-scaled area and the z-score-scaled y-size, and computed the performance using the five abovementioned performance measures (Table III) and compared with the supervised machine learning methods described in the next section (Table IV). In the second way, we adopted the 10,000 times resampling by each time randomly taking the same amount of mRBCs with RETs in the investigated dataset (100 for P1-partition2, 145 for P2 and 103 for P3), and computed the final performance by taking the average precision (Figure 3) of the obtained 10,000 results. Also in this case comparison with the supervised machine learning methods was provided.
+
+### _Other supervised machine learning methods_
+
+The procedure to obtain the results for the supervised analysis was implemented as follows. First of all, a selection of the most important features for the segregation between classes was carried out by means of a machine learning strategy called feature selection. As we did for the learning of the marker threshold in our proposed method (see section G. above), also here we preserved the original count ratio of mRBCs and RETs (10,763/257=41.9) in every cross-validation fold. However, considering that here we learn an entire model and not only a threshold value, we obtained poor results (data not shown) and therefore we moved to adopt a class-balance procedure. In practice, for each machine learning, we trained 10 models.
+
+Figure 1: **Study workflow.** Blood samples were taken from three potential patients and measured using RT-FDC. The output in this case was ten morpho-rheological features together with classification information of each single cell resulting from the fluorescence signal. P1-partion1 was used for training purpose, while P1-partion2, P2 and P3 were used for validation.
+
+Each model was trained with 10-fold cross-validation, and each fold was composed of 257 mRBCs (sampled uniformly at random from the 10,763) and 257 RETs. Then, the average model (obtained by averaging internally the parameter settings of each machine learning) of each machine learning was considered for prediction. Elastic net is a well-known algorithm that can be used for this purpose[21]. It needs a parameter called alpha that combines the L1 and L2 penalties of lasso and ridge regularization methods at different proportions. The alpha value was automatically tuned by changing its value from 0.1 until 0.9 in steps of 0.1 and the one that gave the highest AUC-ROC performance between the two classes (RETs and mRBCs), that is 0.5, was used as a parameter in Elastic net. On the other hand, Gini index [22] is a criterion used as feature selection for random forest (RF) and helps to determine which features are the most important to split the classes of the dataset, by giving them a score depending on how many trees of the random forest they were selected as a split criterion. Another feature selection strategy, and used in this case for Support vector Machine (SVM), is called recursive feature elimination (RFE). It works with the help of an external estimator, in this case SVM, that assigns weights to the features to recursively prune them until a desire number of features is eventually reached. The last feature selection strategy is intrinsically used for partial least square discriminant analysis (PLSDA) and was carried out by calculating the regression coefficients of partial least squares (PLS) and ranking them according to the number of latent variants for PLS.
+
+In order to reduce overfitting in the feature selection, all feature selection algorithms were carried out ten times in a 10-fold cross validation (CV) procedure (a total of 100 iterations). The feature selection consists of two steps. The first step is to compute the final number (of selected features) which is fixed to the average number (that we call _m_) selected for each CV step. The second step is to determine the \(m\) final features to select. This is implemented by assigning to each feature an average weight obtained as the average across the weights gained in the CV steps, and then by selecting the \(m\) features with the highest average weights in the CV steps. Specifically, elastic net selected 7 features (area, aspect ratio, brightness, brightness SD, deformation, inertia ratio and y-size), while Gini index selected 5 (area, area ratio, brightness SD and inertia ratio), as well as RFE (area, area ratio, deformation, inertia ratio and y-size) and PLSDA (aspect, inertia ratio, inertia ratio raw, x-size and y-size).
+
+Once the predictors (features) were chosen, the machine learning models were created in a 10-fold CV step. SVM models (features selected from elastic net and RFE) were produced with the auto optimization of hyperparameters, and with linear and non-linear (RBF) kernels. The RF model (features selected from Gini index) contains five hundred decision trees and was generated with the default parameters (fraction of input data to sample with replacement: 1; minimum number of observations per tree leaf: 1; number of variables to select at random for each decision split: 3 [that is approximatively the square root of the number of variables, which in this case is 10)] as well as PLSDA (default parameter is only the tolerant of convergence: 1E-10) and Logistic Regression (features selection method: elastic net; default parameters are the Model - we used the nominal model - and the Link function - we used logit function).
+
+## III Results
+
+### _Study workflow_
+
+The overall study workflow is represented in Figure 1. The goal of this study is to investigate the ability to classify mature red blood cells (mRBCs) and reticulocytes (RETs) present in the blood of an individual (a patient who needs a diagnosis and could be healthy or pathological), considering only morpho-rheological cell features for the prediction. Hence, we emphasize that the fluorescence probe is used only for testing the performance of the prediction. The first step in the study workflow was to acquire the data from RT-FDC setup (see Methods), which can be used to analyse the presence and prevalence of all major blood cell types, as well as their morpho-rheological features, directly in whole blood[23]. In addition, it can measure the fluorescence intensity of each single cell just as in a conventional flow cytometer. The output is a 2D data matrix, where each row represents a different single cell found in the blood and the columns report for each single cell the respective morpho-rheological values (area, x-size, y-size, etc.) and the corresponding fluorescence intensity that is used to classify cells into mRBCs or RETs. We then proceeded to the unsupervised machine learning by means of PCA using P1-partion1 as training set, with the aim to find the best discriminative dimension by evaluating the separation of the mRBCs and RETs on the first three embedded dimensions. Afterwards, we applied PC-corr algorithm based on the learned best discriminative dimension to detect the discriminative network functional modules that can be used to design the combinatorial marker (because it is a combination of single morpho-rheological markers) for the classification of the two group of cells. To learn the optimal operator point that can be later used for testing, we applied 10-fold cross validation for 10 times to find the combinatorial marker threshold by using P1-partion1. Finally, we tested the performances of our defined combinatorial maker in combination with the learned optimal operator point on three independent datasets, the internal validation dataset P1-partition2 and the external validation datasets P2 and P3 with potential patient validation and cross validation.
+
+### _Unsupervised dimension reduction analysis by PCA and its evaluation_
+
+Due to the natural biological unbalance of mRBCs (90% to 95% of blood cells) against RETs (0.5% to 1.4%) in the blood of healthy adult donors[24], and also to prevent dataset overfitting, we performed the unsupervised learning on the P1-partion1 dataset by using a resampling procedure, which generated from P1-partion1 a total of 10,000 new resampled datasets (see section D of Methodology for detail). The final p-value and AUC-ROC are reported in Table I, and an example PCA results from the 10,000 performed PCA is shown in Supplementary Figure 1, both of which clearly indicate that the second dimension (PC2) of PCA reveals the most significant discrimination, regardless of the measure (p-value or AUC-ROC) used to assess the two-group separation.
+
+Therefore, PC2 weights can be used to apply the PC-corr algorithm, which is able to extract a network composed of feature modules (in this case morpho-rheological measures) related to the two groups separation. We would like to emphasize that the PC-corr algorithm is not a univariate approach that selects single features independently from each other, but instead it is able to perform a multivariate prioritization that emphasizes a cohort of feature-interactions that are most discriminative according to a PCA dimension. This cohort of discriminative feature-interactions generally tends to form one or multiple discriminative network modules that -- as we will illustrate in the following section -- can be used for designing combinatorial markers.
+
+### _PC-corr discriminative module and combinatorial marker design_
+
+We applied the PC-corr algorithm to each of the 10,000 resampled datasets and we considered the mean discriminative network (obtained as mean of 10,000 networks) associated with the PC2 separation. We applied a cut-off of 0.6 on the weights of this mean discriminative network to extract the modules of PC2-related-features. We chose the threshold of 0.6 so that extracted features have at least Pearson Correlation of 0.6 between them, and it is the highest cut-off that ensure the node connectivity, which means, there are only unconnected singular nodes with higher cut-offs (data not shown). We detected a unique discriminative network module composed of three morpho-rheological features (Fig. 2A). The "area" is the cross-sectional area, outlined by the blue contour in Fig. 2B. The "y-size" is the maximal vertical (perpendicular to flow direction) extension of the cell suspended in the liquid passing through the channel, while "x-size" is the maximal horizontal extension of the cell (Fig. 2B). PC-corr also discloses the positive correlations between the discovered features, which are represented by red edges in the network (Fig. 2A). From the ten features available, the PC-corr algorithm helps to unveil those that we should use to design the candidate combinatorial markers. We designed four candidate combinatorial markers, considering the three PC-corr selected and scaled (using z-score transformation) features: the mean of area, y-size and x-size; the mean of area and x-size; and the mean of y-size and x-size. In the next section we will discuss the performance evaluation and validation of these four markers, in comparison to all the original ten morpho-rheological features.
+
+### _Validation of the designed combinatorial markers_
+
+To independently evaluate the classification ability of the four combinatorial markers proposed in the previous section, we considered their performance on the P1-partition2 dataset, which had not been used for learning the markers. The measures used for the evaluation are the Mann-Whitney p-value and AUC-ROC, and the results obtained for each of the four proposed markers and for each of the ten original single features are reported in Table II. Also, in this case we considered the mean performance over 10,000 resampled datasets, containing equal number of mRBCs and RETs (please refer to the methods for details). We discovered that the combination of area and y-size as a unique marker yields the best result. As a comparison, we also calculated the performance offered by the original ten features individually. Taken together, these results prove that a combinatorial selection of the features using PC-corr can tremendously
+
+\begin{table}
+\begin{tabular}{|c|c|c|} \hline
+**PCA** & **mean** & **mean** \\
+**dimension** & **p-value** & **AUC-ROC** \\ \hline PC2 & 2.17E-13 & 0.76 \\ \hline PC1 & 3.67E-03 & 0.61 \\ \hline PC3 & 2.15E-02 & 0.60 \\ \hline \end{tabular}
+\end{table} TABLE I: **Evaluation of Unsupervised Machine Learning Dimension Reduction**
+
+Figure 2: A) Discriminative network module detected by PC-corr and related with PC2 discrimination (cut-off \(=0.6\)). B) Image of a red blood cell flowing in the RT-FDC channel, including illustrations of “area” (bounded by the blue contour), “x-size” and y-size”.
+
+improve the design of the final candidate markers. Interestingly, PC-corr pointed out a discriminative module composed by two interactions between three features: 1) area and y-size; and 2) area and x-size. Our validation in Table II disclosed that area and y-size alone are very discriminative features (AUC: 0.73 and 0.76 respectively), whereas x-size is a poor discriminative feature (AUC: 0.52). The question remains, why PC-corr included also x-size in the discriminative module? The answer is that although singularly x-size is a poor discriminative feature, PC-corr suggests not only _discriminative associations_ between features, but also mechanistic relations between features in the module. In fact, the area is by definition a function of x-size and y-size, and PC-corr successfully infers this from the data independently from the single discriminative power of each feature. This result is possible because PC-corr is a multivariate approach and offers results different from univariate analysis approaches (which test single features), as extensively discussed in the article of Ciucci _et al_. [12].
+
+### _Marker threshold learning and evaluation_
+
+Let us suppose that the morpho-rheological measures of the cell population of a new individual are provided, and that we are interested in applying the combinatorial marker based on the area and y-size (which provided the best performance in the previous evaluation) in order to classify mRBCs and RETs. Yet, what we miss is a threshold for the combinatorial marker so that we can use it to predict new unknown cell's class. In order to learn a proper marker threshold, we used again the P1-partition1 (previously adopted to learn the discriminative module) and selected as best threshold the one that corresponds to the optimal operator point (see section G of Methodology for detail). According to this procedure, we found that 0.51 was the best threshold for the designed marker, computed as the mean of the z-score-scaled area and of the z-score-scaled y-size.
+
+After learning the marker threshold on the P1-partition1, we validated its performance on three independent datasets: P1-partition2, P2 and P3. The rationale is to simulate a real scenario where the cell morpho-rheological features of three new patients (which we called P1-partion2, P2 and P3 and were never used during learning of the marker threshold) were analyzed with our marker. By applying the marker threshold, we computed for each of these potential patients the ability of our marker to predict the true label information (fluorescent probe labels are regarded as ground-truth in this study). In this particular validation, conceptually it does not make sense in our opinion to make a cross-validation, because we are evaluating a real scenario where three patients are going to the doctor and we compute for each of them the performance of our marker in comparison to ground-truth fluorescent probe. The result of this emulation of a realistic clinical estimation are provided in Table III and Supplementary Figure 2, where we display all the main statistics for evaluation of the classification of the cell types (mRBCs vs. RETs) of the three potential patients. However, since it could be also interesting to assess the performance of the investigated markers with 10-fold cross-validation on the 3 validation (patients) independent datasets, these results are provided in Suppl. Table I represented with the average performance on the 10 folds. We found that the overall accuracy on the three datasets is at the level of 0.74 and the overall AUC-ROC is around 0.70 (for P2, it reaches 0.76) with both patient validation and cross validation (Table III and Suppl. Table I). In general, AUC\(<\)0.6 is regarded as poor, while it is considered as acceptable if 0.7\(<\)AUC\(<\)0.8[25]. Therefore, the results here indicate that the designed combinatorial marker (based on area and y-size) together with the learned threshold can offer an acceptable classification performance on the independent validations, both internal and external. On the other hand, we could notice that the level of precision is very low (no higher than 0.05, please refer to Table III last row). This can be seen from the fact that, although the designed marker (and the respective threshold) can achieve an acceptable performance in correctly detecting RETs, on the other hand it makes a relevant false positive error by wrongly classifying a portion of mRBCs as RETs. This portion of wrongly classified mRBCs (which generate false positives) is small in comparison to the total amount of mRBCs, hence the overall specificity is around 0.74 (Table III), which is a relatively good value. However, since the dataset is unbalanced and the fraction of RETs is significantly smaller than mRBCs, even a small fraction of wrongly assigned mRBCs -- since it is much larger than the total RETs - can cause a significant drop in precision. In order to demonstrate that the low precision is only due to the 'over-representation' of mRBCs and that the marker and threshold inferred are valid, we repeated the same validation analysis done in Table 3 considering mRBCs sampled at random in an equal amount to RETs (see Methods for details). The results reported in Figure 3 demonstrate that if, in the independent validation phase, we reduce the naturally occurring over-representation of mRBCs -- using a procedure that is not biased, since it exploits a class-balance procedure based on random uniform mRBC sampling -- then the level of precision increases drastically. Indeed, PC-corr marker increases precision from less than 0.05 (Table 3, last line) to more than 0.82 (Figure 3, first bar in each plot). This shows that the low levels of precision do not originate from a learning error of the combinatorial marker and threshold, but from the over-representation (which can be interpreted as a sort of 'oversampling') of mRBCs in comparison to RETs. In practice, the low precision is generated by the fact that mRBCs are more abundantly represented in the dataset than RETs. This implies that, although the threshold is correct, the amount of mRBCs that pass the threshold assuming values similar to RETs is minor in comparison to the total amount of mRBCs. But it is still remarkable in comparison to the few total RETs present in the dataset. Taken together these results suggest that we mainly demonstrate the validity of the proposed unsupervised analysis pipeline as a proof of concept. For real-world application, higher level of precision is required, and the problem of unbalanced cell's cohorts should be adequately addressed in future studies.
+
+Finally, since the method used as reference for ab-initio labelling of the cells is based on fluorescence, we cannot assert that the mRBCs that pass the threshold assuming values of the proposed morpho-rheological marker similar to RETs are in general incorrectly assigned. In fact, to be more correct, we can only assert that there is a disagreement between our morpho-rheological marker assignment and the fluorescent assignment. Therefore, we can speculate that these mRBC cells, which are RET-like according to our morpho-rheological marker and not-RET-like according to fluorescence, should be investigated with more attention in future studies, because they might hide a cell sub-population in a 'gray area' that lies between mature red blood cells (mRBCs) and reticulocytes (RETs), which are immature red blood cells. A dichotomic separation between mRBCs and RETs might be over-simplistic, and a more truthful cell-phenotype landscape might consist of a fuzzy scenario populated also by intermediate and transition states. In fact, modern blood counters do distinguish different subpopulations of reticulocytes by their level of fluorescence. However, for the given measurements with the given gates, total reticulocyte numbers are in agreement with standard blood count performed at the university hospital.
+
+Figure 3: Mean precision performance of evaluated methods for an independent validation class-balanced scenario using 10,000 permutations of random uniform mRBC sampling. The bar color is associated with the number of features used to train the respective model. Light blue used two features; gray used five features; blue used seven features. The red whiskers report the standard deviation. A) Performance in internal validation P1-partition 2 data. B) Performance in external validation P2 data. C) Performance in external validation P3 data.
+
+### _Performance comparison with supervised approach_
+
+The motivation for this unsupervised approach is the fact that the data are highly unbalanced, with 10,763 mRBCs (negative class) versus 257 RETs (positive class) for the machine learning model generation. It is known that regular supervised methods do not work well in these scenarios because they tend to predict new samples as the majority class in the training set, since these models try to optimize by accuracy. Moreover, Smialowski and colleagues demonstrated that PCA-based feature selection was more robust and less prone to overfitting in their study [17]. However, in order to quantitatively evaluate the extent to which our approach (which is based on unsupervised learning in the first part) offers better results, we compared it with the following supervised machine learning methods (which were trained according to the class-balance strategy reported in the methods section H): Support Vector Machine (SVM), Linear Regression (LR), Random Forest (RF) and Partial Least Squares Discriminant Analysis (PLSDA) using different feature selection strategies such as Gini index (for RF), Recursive Feature Elimination (for SVM) and elastic net (for LR and SVM). In addition, Elastic Net was also used as an all-in-one feature selection and classification method. Thus, we compared in total 6 supervised models versus our approach.
+
+The results which show the performance in validation of the other MLs on three independent datasets is reported in Suppl. Table II, which should be compared with Table III for PC-corr. In order to simplify this investigation, we created a Table IV that summarizes the contrast between PC-corr combinatorial marker (based only on the average of two measures) and the other MLs combinatorial markers (based in general on models that adopt from 5 to 7 features, according to supervised feature selection). The comparison consists in counting how many times PC-corr performs better than the other methods considering 5 different evaluation measures in 3 independent validation sets (15 evaluations in total). Remarkably, from Table IV emerges that PC-corr using only two features provided a performance comparable and often higher (in 5 of the 6 comparisons) than more complicated models based on different machine learning rationales which use 5 to 7 features. In addition, in Figure 3 we compare the precision increase of our method versus the increase of the other MLs, when the validation on the three independent datasets is balanced by random uniform mRBC sampling. The extensive comparison provided here demonstrates that our proposed method is well performing in this challenging classification task also in comparison to state-of-the-art supervised methods. Taken together, the advantage of PC-corr is twofold: (i) it offers, using only two features, comparable performance to state-of-the-art methods that need from 5 to 7 features; (ii) it provides remarkable higher precision performance in comparison to state-of-the-art methods (Figure 3) when the datasets are balanced by random uniform mRBC sampling.
+
+**Table IV**
+
+**Comparison Of Performance Between PC-corr Based Marker Vs Other Machine Learning Based Markers On Three Independent Datasets.** The first column indicates the ML-based combinatorial markers based on the number of features indicated in brackets. The second column indicates the number of cases in which (comparing Table III of PC-corr validation with the respective tables of the ML-methods reported in Suppl. Table II) ML-methods perform better than PC-corr. The third column indicates the number of cases in which PC-corr (whose combinatorial marker is based on two features, which is the value reported in brackets near PC-corr name) performs better than other MLs. The fourth column reports the number of cases that are tied. Bold characters emphasize the number of times that a method performs better than PC-corr or vice versa. Remarkably, PC-corr performs better in 5 of the 6 comparisons.
+
+## IV Discussion
+
+RT-FDC is a powerful microfluidic technique[9] for the morpho-rheological characterization of cells and its correlation with conventional fluorescence-based analysis. Its high-throughput capability allows for efficient measurements also in cases with scarce populations such as reticulocytes. In this study, we demonstrated that the morpho-rheological features obtained with RT-FDC can be exploited to develop promising label-free combinatorial markers for cell biology research. First, we proposed a general computational and unsupervised machine learning framework for the design of combinatorial morpho-rheological markers and the marker-threshold definition. Our aim was to explore the potential and limitations of using a basic unsupervised and linear approach. We were interested in defining a baseline that could suggest what is possible to achieve using a simple and easily interpretable combination of morpho-rheological features to design a combinatorial marker for direct cell classification.
+
+Our computational framework was proven in multiple independent validations to be able to provide acceptable performance when applied to a challenging (unbalanced-dataset) classification task such as the one to classify mRBCs vs RETs. This result is very promising and we hope that future studies investigate and address the current limitation of the methodology. Despite the acceptable level of classification, RETs are detected with low precision which, in combination with their naturally low prevalence, is problematic for some real-world applications. A significant number of cells, classified as mRBCs by their lack of RNA content staining, are falsely assigned to the RET population. This disagreement between the morpho-rheological and the fluorescence-basedcell assignment, needs further investigation because it might indicate an error of detection. More interestingly, it could also suggest the presence of hidden, uncategorized sub-populations of cells.
+
+Although the methods and findings provided here need further investigation to be used in clinical applications, they demonstrate the predictive potential of morpho-rheological phenotyping for computational-driven cell characterization/classification. Therefore, we expect that this study could contribute to the definition of new standards of analysis in precision and systems biomedicine.
+
+Since this study was intended to evaluate how to implement a basic unsupervised and linear approach to discriminate mature RBCs and reticulocytes in the blood of an individual by using morpho-rheological phenotype data obtained from RT-FDC, future studies might go beyond this and investigate: i) more advanced approaches based on nonlinear machine learning directly on morpho-rheological data; and ii) deep learning techniques applied directly on the image samples, which could improve the performance of the classification without the pre-processing step to extract morpho-rheological features from the images.
+
+## V Conclusion
+
+We propose an interdisciplinary study that deals with cell labeling from a different perspective by combining biophysics with machine intelligence tools. We started with a newly developed high-throughput single cell mechanics measurement technology, named real-time deformability and fluorescence cytometry (RT-FDC), and then we applied unsupervised machine learning to predict the labels of single cells, in particular we consider the task to discriminate and classify mature red blood cells against reticulocytes, which are immature red blood cell. We focus our study on this specific task because the investigation of reticulocytes (their percentage and cellular features) in the blood is important to quantitatively evaluate conditions that affect RBCs, such as anemia or bone marrow disorders. Our results suggest that the proposed machine intelligence data-driven methodology can provide promising results for the morpho-rheological-based prediction of red blood cells, therefore it can point out a new complementary direction to fluorescent cell labeling.
+
+## Author Contribution
+
+CVC and JG conceived the study. CVC and YG designed the computational procedure. PR and JG devised and carried out the RT-FDC experiments. NT was responsible for acquisition of the blood samples. YG performed the computational analysis with CVC help. YG realized the MATLAB code and SC tested for quality control. CD realized the MATLAB code and performed the computational analysis for the supervised machine learning part. YG and CVC designed the figures and tables. YG and CVC wrote the article with inputs and corrections from all the other authors. YG and SC realized the figures under the CVC guidance. JG led, directed and supervised the study for the cell mechanics part. CVC led, directed and supervised the study for the computational part.
+
+## Additional Information
+
+Competing financial interests
+
+PR is holding shares of Zellmechanik Dresden GmbH, that sells devices based on RT-DC.
+
+## Acknowledgment
+
+We thank Alejandro Rivera Prieto and Salvatore Girardo, head of the Microstructure Facility of the Center for Molecular and Cellular Bioengineering (CMCB) at the Technische Universitat Dresden (in part funded by the Sachsisches Ministerium fur Wissenschaft und Kunst and the European Fund for Regional Development) for help with microfluidic chip production.
+
+_Funding:_ The work in C.V.C. research group was mainly supported by the independent group leader starting grant of the BIOTEC at Technische Universitat Dresden (TUD) and by the Klaus Tschira Stiftung (KTS) gGmbH, Germany, grant name: Lipidom Signaturen fuer Alzheimer und Multiple Sklerose (Code: 00.285.2016). S.C. wrap-up postdoc period was supported by the Dresden International Graduate School for Biomedicine and Bioengineering (DIGS-BB), granted by the Deutsche Forschungsgemein-schaft (DFG) in the context of the Excellence Initiative. C. D. is supported by the Research Grant - Doctoral Programs in Germany from the Deutscher Akademischer Austauschdienst (DAAD), Promotion program Nr: 57299294. We acknowledge support by the German Research Foundation and the Open Access Publication Funds of the TU Dresden. Financial support from the Alexander-von-Humboldt Stiftung (Humboldt-Professorship to J.G.) and the DFG KFO249 (GU 612/2-2 grant to J.G.) is gratefully acknowledged.
+
+## References
+
+* [1] F. Hans and S. Dimitrov, "Histone H3 phosphorylation and cell division.," _Oncogene_, vol. 20, no. 24, pp. 3021-3027, 2001.
+* [2] B. J. Belin, L. M. Goins, and R. D. Mullins, "Comparative analysis of tools for live cell imaging of actin network architecture," _Bioarchitecture_, vol. 4, no. 6, pp. 189-202, 2014.
+* [3] A. Sakaue-Sawano _et al._, "Visualizing Spatiotemporal Dynamics of Multicellular Cell-Cycle Progression," _Cell_, vol. 132, no. 3, pp. 487-498, 2008.
+* [4] a Y. Chen, C. Yu, B. Gatto, and L. F. Liu, "DNA minor groove-binding ligands: a different class of mammalian DNA topoisomerase I inhibitors.," _Proc. Natl. Acad. Sci. U. S. A._, vol. 90, no. 17, pp. 8131-8135, 1993.
+* Influence on chromatin organization and histone
+interactions," _Cytom. Part A_, vol. 73, no. 6, pp. 555-562, 2008.
+* [6] E. C. Jensen, "Use of fluorescent probes: their effect on cell biology and limitations," _Anat. Rec._, vol. 295, no. 12, pp. 2031-2036, 2012.
+* [7] D. R. Gossett _et al._, "Hydrodynamic stretching of single cells for large population mechanical phenotyping," _Proc. Natl. Acad. Sci._, vol. 109, no. 20, pp. 7630-7635, 2012.
+* [8] E. Moenardbary and A. R. Harris, "Cell mechanics: Principles, practices, and prospects," _Wiley Interdiscip. Rev. Syst. Biol. Med._, vol. 6, no. 5, pp. 371-388, 2014.
+* [9] P. Rosendahl _et al._, "Real-time fluorescence and deformability cytometry," _Nat. Methods_, 2018.
+* [10] C. Brugnara, "Use of reticulocyte cellular indices in the diagnosis and treatment of hematological disorders," _Int. J. Clin. Lab. Res._, vol. 28, no. 1, pp. 1-11, 1998.
+* [11] R. E. Waugh, A. Mantalaris, R. G. Bauserman, W. C. Hwang, and J. H. D. Wu, "Membrane instability in late-stage erythropoiesis," _Blood_, vol. 97, no. 6, pp. 1869-1875, 2001.
+* [12] S. Ciucci _et al._, "Enlightening discriminative network functional modules behind Principal Component Analysis separation in differential-omic science studies," _Sci. Rep._, vol. 7, p. 439-46, Mar. 2017.
+* [13] B. H. Davis, K. Ornvold, and N. C. Bigelow, "Flow cytometric reticulocyte maturity index: A useful laboratory parameter of erythropoietin activity in anemia," _Cytometry_, vol. 22, no. 1, pp. 35-39, 1995.
+* [14] M. Herbig, M. Krater, K. Plak, P. Muller, J. Guck, and O. Otto, "Real-time deformability cytometry: Label-free functional characterization of cells," in _Methods in Molecular Biology_, 2018.
+* [15] O. Otto _et al._, "Real-time deformability cytometry: on-the-fly cell mechanical phenotyping.," _Nat. Methods_, vol. 12, no. 3, p. 199-202, 4 p following 202, 2015.
+* [16] G. Bradski, "The OpenCV Library," _Dr. Dobb's J. Softw. Tools_, 2000.
+* [17] P. Smialowski, D. Frishman, and S. Kramer, "Pitfalls of supervised feature selection," _Bioinformatics_, vol. 26, no. 3, pp. 440-443, Feb. 2010.
+* [18] C. V. Cannistraci, T. Ravasi, F. M. Montevecchi, T. Ideker, and M. Alessio, "Nonlinear dimension reduction and clustering by Minimum Curvilinearity unfold neuropathic pain and tissue embryological classes.," _Bioinformatics_, vol. 26, no. 18, pp. i531-9, Sep. 2010.
+* [19] H. He and E. A. Garcia, "Learning from imbalanced data," _IEEE Trans. Knowl. Data Eng._, vol. 21, no. 9, pp. 1263-1284, 2009.
+* [20] H. B. Mann and D. R. Whitney, "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other," _Ann. Math. Stat._, vol. 18, no. 1, pp. 50-60, 1947.
+* [21] H. Zou and T. Hastie, "Regularization and variable selection via the elastic net," _J. R. Stat. Soc. Ser. B (statistical Methodol._, vol. 67, no. 2, pp. 301-320, 2005.
+* [22] L. Breiman, "(34) Classification and regression trees Regression trees," _Encycl. Ecol._, 1984.
+* [23] N. Toepfner _et al._, "Detection of human disease conditions by single-cell morpho-rheological phenotyping of blood," _Elife_, vol. 7, 2018.
+* [24] G. d'Onofrio, R. Chirillo, G. Zini, G. Caenaro, M. Tommasi, and G. Micciulli, "Simultaneous Measurement of Reticulocyte and Red Blood Cell Indices in Healthy Subjects and Patients With Microocytic and Macroocytic Anemia," _Blood_, vol. 85, no. 3, pp. 818-823, 1995.
+* [25] E. Dubois and K. Pohl, _Advanced Information Systems Engineering: 29th International Conference, CAiSE 2017, Essen, Germany, June 12-16, 2017, Proceedings_, vol. 10253. Springer, 2017.
+
+\\ \end{tabular} \begin{tabular}{c c}  & Yan Ge received the B.Sc. degree in biotechnology from the Southwest University of Science and Technology, China, in 2007, and the PhD degree in China Agricultural University in 2012. He was postdoc at the Biotechnology Center (BIOTEC) of Technische Universitat Dresden from 2015 to 2018, under the joint supervision of Dr. Carlo Vittorio Cannistraci and Dr. Jochen Guck. Now he works as postdoc in the Medical Theoratical Center (MTZ) of Universitatsklinikum Carl Gustav Carus Dresden. His research interests include single cell omics, machine learning and computational immunology. \\ \end{tabular} \begin{tabular}{c c}  & Philipp Rosendahl, received his Diploma and Ph.D. degree in Physics from Technische Universitat Dresden, Germany in 2013 and 2018, respectively. At the Biotechnology Center (BIOTEC) he was developing the methods RT-(F)DC and applying them to biological questions under supervision of Prof. Jochen Guck (Nature methods 2015 & 2018). He is now Head of Product Development and co-founder of the company _Zellmechanik Dresden GmbH_, which is making RT-DC commercially available. His research focusses on the technical development of microfluidic real-time systems for cytometry applications. Dr. Rosendahl was awarded with the _Georg-Helm-Prize_ for his Diploma thesis in 2013, the _Klaus-Goertller-Prize_ for his dissertation and the _IBA Best-Paper award_ for the RT-FDC method publication in 2018. \\ \end{tabular} \begin{tabular}{c c}  & Claudio Duran received the Engineer diploma degree in bioinformatics from the University of Talca, Chile, in 2017. He is currently a PhD student in the Biomedical Cybernetics lab led by Dr. Carlo Vittorio Cannistraci in Technische Universitat Dresden. His research interests include machine learning, network science and systems biomedicine. \\ \end{tabular} \begin{tabular}{c c}  & Nicole Toepfner received her MD degree and approach at the Heinrich-Heine University Dusseldorf, Germany and specialized in Pediatrics at the University Medical Centers of Freiburg (2008-2012) and Dresden (2012-2014), Germany. She was a scholarship student of the DFG graduate school 320, an IFMA scholarship holder (2006) and a DFG Gerok fellow (2014). Her work on streptococcal infections was awarded by the Young Investigator Award of the German Society for Pediatric Infectious Disease (2013). Before her specialist training in Pediatric Hematology and Oncology, Nicole Toepfner was a postdoc in the group of Prof. Jochen Guck, BIOTEC Dresden, Germany and of Prof. Edwin R. Chilvers, University of Cambridge, UK (2014-2016). She developed a label-free human blood cell analysis capable of detecting functional immune cell changes (e.g. of post primed neutrophils) in infection and inflammation (Elife 2018, Frontiers in Immunology 2018, Journal of Leukocyte Biology 2019). \\ \end{tabular} \begin{tabular}{c c}  & Sara Ciucci received the B.Sc. degree in Mathematics from the University of Padova, Italy, in 2010, the M.Sc. degree in Mathematics from the University of Trento, Italy, in 2014, and the PhD degree in Phynisics from Technische Universitat Dresden, Germany, in 2018, at the Biomedical Cybernetics lab under the supervision of Dr. Carlo Vittorio Cannistraci. She is currently a postdoctoral researcher at the Biotechnology Center (BIOTEC) of Technische Universitat Dresden in the Biomedical Cybernetics lab. Her research interests include machine learning, network science and systems biomedicine. \\ \end{tabular} 
+\begin{tabular}{c c}  & Jochen Guck is a cell biophysicist, born in Germany in 1973. He obtained his PhD in Physics from the University of Texas at Austin in 2001. After being a group leader at the University of Leipzig, he moved to the Cavendish Laboratory at Cambridge University as a Lecturer in 2007 and was promoted to Reader in 2009. In 2012 he became Professor of Cellular Machines at the Biotechnology Center of the Technische Universitat Dresden. As of October 2019 he is now Director at the Max Planck Institute for the Science of Light and the Max-Planck-Zentrum fur Physik und Medizin in Erlangen, Germany. His research centers on exploring the physical properties of biological cells and tissues and their importance for their function and behavior. He also develops novel photonic, microfluidic and scanning-force probe techniques for the study of these optical and mechanical properties. The ultimate goal is utilizing this insight for novel diagnostic and therapeutic approaches. He has authored over 100 peer-reviewed publications and four patents. His work has been recognized by several awards, amongst them the Cozzarelli Award in 2008, the Paterson Prize in 2011 and an Alexander-von-Humboldt Professorship in 2012.
+
+Footnote 1: [https://www.ieee.org/publications_standards/publications/rights/index.html](https://www.ieee.org/publications_standards/publications/rights/index.html)
+
+Carlo Vittorio Cannistraci is a theoretical engineer and was born in Milazzo, Sicily, Italy in 1976. He received the M.S. degree in Biomedical Engineering from the Polytechnic of Milano, Italy, in 2005 and the Ph.D. degree in Biomedical Engineering from the Inter-polytechnic School of Doctorate, Italy, in 2010. From 2009 to 2010, he was visiting scholar in the Integrative Systems Biology lab of Dr. Trey Ideker at the University California San Diego (UCSD), CA, USA. From 2010 to 2013, he was postdoc and then research scientist in machine intelligence and complex network science for personalized biomedicine at the King Abdullah University of Science and Technology (KAUST), Saudi Arabia. Since 2014, he has been Independent Group Leader and Head of the Biomedical Cybernetics lab at the biotechnological Center (BIOTEC) of the TU-Dresden, Germany. He is also affiliated with the MPI Center for Systems Biology Dresden and with the Tsinghua Laboratory of Brain and Intelligence. He is author of three book chapters and more than 40 articles. His research interests include subjects at the interface between physics of complex systems, complex networks and machine learning theory, with particular interest for applications in biomedicine and neuroscience.
+
+Dr. Cannistraci is member of the Network Science Society, member of the International Society in Computational Biology, member of the American Heart Association, member of the Functional Annotation of the Mammalian Genome Consortium. He is an Editor for the mathematical physics board of the journal Scientific Reports edited by Nature and of PLOS ONE. _Nature Biotechnology_ selected his article (_Cell_ 2010) on machine learning in developmental biology to be nominated in the list of 2010 notable breakthroughs in computational biology. _Circulation Research_ featured his work (_Circulation Research_ 2012) on leveraging a cardiovascular systems biology strategy to predict future outcomes in heart attacks, commenting: "a space-aged evaluation using computational biology". In 2017, Springer-Nature scientific blog highlighted with an interview to Dr. Cannistraci his recent study on "How the brain handles pain through the lens of network science". In 2018, the American Heart Association covered on its website Dr. Cannistraci's chronobiology discovery on how the _sunshine affects the risk and time onset of heart attack_. The TU-Dresden honoured Dr. Cannistraci of the _Young Investigator Award 2016 in Physics_ for his recent work on the local-community-paradigm theory and link prediction in monopartite and bipartite complex networks.

2003.00011v1.mmd

@@ -0,0 +1,191 @@
+[MISSING_PAGE_FAIL:1]
+
+Finding the ground state of (i.e., "solving") such systems is interesting from the perspective of thermodynamics, as one can observe phenomena such as phase transitions [5; 6], but also practically useful as discrete optimization problems can be mapped to spin-glass models (e.g., the travelling salesperson problem or the knapsack problem) [7]. The Metropolis-Hastings algorithm [8; 9] can be used to simulate the spin glass at arbitrary temperature; thus, it is used ubiquitously for SA. By beginning the simulation at a high temperature, one can slowly cool the system over time, providing sufficient thermal energy to escape local minima, and arrive at the ground state "solution" to the problem. The challenge is to find a temperature schedule that minimizes computational effort while still arriving at a satisfactory solution; if the temperature is reduced too rapidly, the system will become trapped in a local minimum, and reducing the temperature too slowly results in an unnecessary computational expense. Kirkpatrick et al. [1; 10] proposed starting at a temperature that results in an 80% acceptance ratio (i.e., 80% of Metropolis spin flips are accepted) and reducing the temperature geometrically. They also recommended monitoring the objective function and reducing the cooling rate if the objective value (e.g., the energy) drops too quickly. More-sophisticated adaptive temperature schedules have been investigated [11]. Nevertheless, in his 1987 paper, Bounds [12] said that "choosing an annealing schedule for practical purposes is still something of a black art".
+
+When framed in the advent of quantum computation and quantum control, establishing robust and dynamic scheduling of control parameters becomes even more relevant. For example, the same optimization problems that can be cast as classical spin glasses are also amenable to quantum annealing [13; 14; 15; 16; 17], exploiting, in lieu of thermal fluctuations, the phenomenon of quantum tunnelling [18; 19; 20] to escape local minima. Quantum annealing (QA) was proposed by Finnila et al. [21] and Kadowaki and Nishimori [22], and, in recent years, physical realizations of devices capable of performing QA (quantum annealers), have been developed [23; 24; 25; 26], and are being rapidly commercialized. As these technologies progress and become more commercially viable, practical applications [17; 27] will continue to be identified and resource scarcity will spur the already extant discussion of the efficient use of annealing hardware [28; 29].
+
+Nonetheless, there are still instances where the classical (SA) outperforms the quantum (QA) [30], and improving the former should not be undervalued. _In silico_ and hardware annealing solutions such as Fujitsu's FPGA-based Digital Annealer [31], NTT's laser-pumped coherent Ising machine (CIM) [32], and the quantum circuit model algorithm known as QAOA [33; 34] all demand the scheduling of control parameters, whether it is the temperature in the case of the Digital Annealer, or the power of the laser pump in the case of CIM. Heuristic methods based on trial-and-error experiments are commonly used to schedule these control parameters, and an automatic approach could expedite development, and improve the stability of such techniques.
+
+In this work, we demonstrate the use of a reinforcement learning (RL) method to learn the "black art" of classic SA temperature scheduling, and show that an RL agent is able to learn dynamic control parameter schedules for various problem Hamiltonians. The schedules that the RL agent produces are dynamic and reactive, adjusting to the current observations of the system to reach the ground state quickly and consistently without _a priori_ knowledge of a given Hamiltonian. Our technique, aside from being directly useful for _in silico_ simulation, is an important milestone for future work in quantum information processing, including for hardware- and software-based control problems.
+
+Figure 1: Two classes of Hamiltonian problems are depicted. (a) The weak-strong clusters (WSC) model comprises two bipartite clusters. The left cluster is biased upward; the right cluster is biased downward. All couplings are equal and of unit magnitude. The two clusters are coupled via the eight central nodes. This model exhibits a deep local minimum very close in energy to the model’s global minimum. When initialized in the local minimum, the RL agent is able to learn schemes to escape the local minimum and arrive at the global minimum, without any explicit knowledge of the Hamiltonian. (b) Here we present an example spin-glass model. The nodes are coupled to nearest neighbours with random Gaussian-distributed coupling coefficients. The nodes are unbiased, and the couplings are changed at each instantiation of the model. The RL algorithm is able to learn a dynamic temperature schedule by observing the system throughout the annealing process, without explicit knowledge of the form of the Hamiltonian, and the learned policy can be applied to all instances of randomly generated couplings. We demonstrate this on variably sized spin glasses and investigate the scaling with respect to a classic linear SA schedule. In (c), we show snapshots of a sample progression of a configuration undergoing SA under the ferromagnetic Ising model Hamiltonian and a constant cooling schedule. The terminal state, all spins-up, is the ground state; this anneal would be considered successful.
+
+## II Reinforcement Learning
+
+Reinforcement learning is a branch of dynamic programming whereby an agent, residing in state \(s_{t}\) at time \(t\), learns to take an action \(a_{t}\) that maximizes a cumulative reward signal \(R\) by dynamically interacting with an environment [35]. Through the training process, the agent arrives at a policy \(\pi\) that depends on some observation (or "state") of the system, \(s\). In recent years, neural networks have taken over as the _de facto_ function approximator for the policy. Deep reinforcement learning has seen unprecedented success, achieving superhuman performance in a variety of video games [36; 37; 38; 39], board games [40; 41; 42], and other puzzles [43; 44]. While many reinforcement learning algorithms exist, we have chosen to use proximal policy optimization (PPO) [45], implemented within Stable Baselines [46] for its competitive performance on problems with continuous action spaces.
+
+## III The Environment
+
+We developed an OpenAI gym [47] environment which serves as the interface to the "game" of simulated annealing. Let us now define some terminology and parameters important to simulated annealing. For a given Hamiltonian, defining the interactions of \(L\) spins, we create \(N_{\text{reps}}\) randomly initialized replicas (unless otherwise specified). The initial spins of each replica are drawn from a Bernoulli distribution with probability of a spin-up being randomly drawn from a uniform distribution. These independent replicas are annealed in parallel. The replicas follow an identical temperature schedule with their uncoupled nature providing a mechanism for statistics of the system to be represented through an ensemble of measurements. In the context of the Metropolis-Hastings framework, we define one "sweep" to be \(L\) random spin flips (per replica), and one "step" to be \(N_{\text{sweeps}}\). After every step, the environment returns an observation of the current state \(s_{t}\) of the system, an \(N_{\text{reps}}\times L\) array consisting of the binary spin values present. This observation can be used to make an informed decision of the action \(a_{t}\) that should be taken. The action, a single scalar value, corresponds to the total inverse temperature change \(\Delta\beta\) that should be carried out over the subsequent step. The choice of action is provided to the environment, and the process repeats until \(N_{\text{steps}}\) steps have been taken, comprising one full anneal, or "episode" in the language of RL. If the chosen action would result in the temperature becoming negative, no change is made to the temperature and the system continues to evolve under the previous temperature.
+
+### Observations
+
+For the classical version of the problem, an observation consists of the explicit spins of an ensemble of replicas. In the case of an unknown Hamiltonian, the ensemble measurement is important as the instantaneous state of a single replica does not provide sufficient information about the current temperature of the system. Providing the agent with multiple replicas allows it to compute statistics and have the possibility of inferring the temperature. For example, if there is considerable variation among replicas, then the system is likely hot, whereas if most replicas look the same, the system is probably cool.
+
+When discussing a quantum system, where the spins represent qubits, direct mid-anneal measurement of the system is not possible as measurement causes a collapse of the wavefunction. To address this, we discuss experi
+
+Figure 2: A neural network is used to learn the control parameters for several SA experiments. By observing a lattice of spins, the neural network can learn to control the temperature of the system in a dynamic fashion, annealing the system to the ground state. The spins at time \(t\) form the state \(s_{t}\) fed into the network. Two concurrent convolutional layers extract features from the state. These features are combined with a dense layer and fed into a recurrent module (an LSTM module) capable of capturing temporal characteristics. The LSTM module output is reduced to two parameters used to form the policy distribution \(\pi_{\theta}(a_{t}\mid s_{t})\) as well as to approximate the value function \(V(s_{t})\) used for the generalized advantage estimate.
+
+ments conducted in a "destructive observation" environment, where measurement of the spins is treated as a "one-time" opportunity for inclusion in RL training data. The subsequent observation is then based on a different set of replicas that have evolved through the same schedule, but from different initializations.
+
+## IV Reinforcement learning architecture
+
+Through the framework of reinforcement learning, we wish to produce a policy function \(\pi_{\theta}(a_{t}\mid s_{t})\) that takes the observed binary spin state \(s_{t}\in\{-1,1\}^{N_{\text{rep}}\times L}\) and produces an action \(a_{t}\) corresponding to the optimal change in the inverse temperature. Here \(\pi\) is a distribution represented by a neural network and the subscript \(\theta\) denotes parameterization by learnable weights \(\theta\). We define the function
+
+\[\phi_{k}(s_{t})\in\{-1,1\}^{1\times L}\]
+
+as an indexing function that returns the binary spin values for the \(k\)-th rep of state \(s_{t}\).
+
+The neural network is composed of two parts: a convolutional feature extractor, and a recurrent network to capture the temporal characteristics of the problem. The feature extractor comprises two parallel two-dimensional convolutional layers. The first convolutional layer has \(N_{k_{r}}\) kernels of size \(1\times L\), and aggregates along the replicas dimension, enabling the collection of spin-wise statistics across the replicas. The second convolutional layer has \(N_{k_{s}}\) kernels of size \(N_{\text{reps}}\times 1\) and slides along the spin dimension, enabling the aggregation of replica-wise statistics across the spins. The outputs of these layers are flattened, concatenated, and fed into a dense layer of size \(N_{d}\) hidden nodes. This operates as a latent space encoding for input to a recurrent neural network (a long short-term memory, or LSTM, module [48]), used to capture the sequential nature of our application. The latent output of the LSTM module is of size \(N_{L}\). For simplicity, we set \(N_{k_{r}}=N_{k_{s}}=N_{d}=N_{L}=64\). All activation functions are hyperbolic tangent (tanh) activations. Since \(a_{t}\) can assume a continuum of real values, this task is referred to as having a continuous action space, and thus standard practice is for the network to output two values corresponding to the first and second moments of a normal distribution. During training, when exploration is desired, an entropic regularization in the PPO cost function can be used to encourage a high variance (i.e., encouraging \(\sigma^{2}\) to remain sizable). Additionally, PPO requires an estimate of the generalized advantage function [49], the difference between the reward received by taking action \(a_{t}\) while in state \(s_{t}\), and the expected value of the cumulative future reward prior to an action being taken. The latter, termed the "value function", or \(V(s_{t})\), cannot possibly be computed because we know only the reward from the action that was chosen, and nothing about the actions that were not chosen, but we can estimate it using a third output from our neural network. Thus, as seen in Figure 2, the neural network in this work takes the state \(s_{t}\) and maps it to three scalar quantities, \(\mu\), \(\sigma^{2}\), and \(V(s_{t})\), defining the two moments of a normal distribution and an estimate of the value function, respectively. At the core of RL is the concept of reward engineering, that is, developing a reward scheme to inject a notion of success into the system. As we only care about reaching the ground state by the end of a given episode, we use a sparse reward scheme, with a reward of zero for every time step before the terminal step, and a reward equal to the negative of the minimum energy as the reward for the terminal step, that is,
+
+\[R_{t}=\begin{cases}0,&t<N_{\text{steps}}\\ -\min_{k}\mathcal{H}(\phi_{k}(s_{t})),&t=N_{\text{steps}}\end{cases}, \tag{1}\]
+
+where \(k\in[1,N_{\text{reps}}]\). With this reward scheme, we encourage the agent to arrive at the lowest possible energy by the time the episode terminates, without regard to what it does in the interim. In searching for the ground state, the end justifies the means.
+
+When optimizing the neural network, we use a PPO discount factor of \(\gamma=0.99\), eight episodes between weight updates, a value function coefficient of \(c_{1}=0.5\), an entropy coefficient of \(c_{2}=0.001\), a clip range of \(\epsilon=0.05\), a learning rate of \(\alpha=1\times 10^{-6}\), and a single minibatch per update. Each agent is trained over the course of \(25,000\) episodes (anneals), with \(N_{\text{steps}}=40\) steps per episode, and with \(N_{\text{sweeps}}=100\) sweeps separating each observation. We used \(N_{\text{reps}}=64\) replicas for each observation.
+
+## V Evaluation
+
+Whereas the RL policy can be made deterministic, meaning a given state always produces the same action, the underlying Metropolis algorithm is stochastic; thus, we must statistically define the metric for success. We borrow this evaluation scheme from Aramon et al. [50]. Each RL episode will either result in "success" or "failure". Let us define the "time to solution" as
+
+\[T_{s}=\tau n_{99}\,, \tag{2}\]
+
+that is, the number of episodes that must be run to be \(99\%\) sure the ground state has been observed at least one time (\(n_{99}\)), multiplied by the time \(\tau\) taken for one episode.
+
+Let us also define \(X_{i}\) as the binary outcome of the \(i\)-th episode, with \(X_{i}=1\) (0) if at least one (none) of the \(N_{\text{reps}}\) replicas are observed to be in the ground state at episode termination. The quantity \(Y\equiv\sum_{i=1}^{n}X_{i}\) is the number of successful episodes after a total of \(n\) episodes, and \(p\equiv P(X_{i}=1)\) denotes the probability that an anneal \(i\) will be successful. Thus the probability of exactly \(k\) out of \(n\) episodes succeeding is given by the probability mass function of the binomial distribution
+
+\[P(Y=k\mid n,p)=\binom{n}{k}\,p^{k}(1-p)^{n-k}. \tag{3}\]
+
+To compute the time to solution, our quantity of interest is the number of episodes \(n_{99}\) where \(P=0.99\), that is,
+
+\[P(Y\geq 1\mid n_{99},p)=0.99.\]
+
+From this and (3), it can be shown that
+
+\[n_{99}=\frac{\log{(1-0.99)}}{\log{(1-p)}}.\]
+
+In the work of Aramon et al. [50], \(p\) is estimated using Bayesian inference due to their large system sizes sometimes resulting in zero successes, precluding the direct calculation of \(p\). In our case, to evaluate a policy, we perform 100 runs for each of 100 instances and compute \(p\) directly from the ratio of successful to total episodes, that is, \(p=\hat{X}\).
+
+## VI Hamiltonians
+
+We present an analysis of two classes of Hamiltonians. The first, which we call the weak-strong clusters model (WSC; see Figure 1a), is an \(L=16\) bipartite graph with two fully connected clusters, inspired by the "Chimera" structure used in D-Wave Systems' quantum annealing hardware [51]. In our case, one cluster is negatively biased with \(h_{i}=-0.44\) and the other positively biased with \(h_{i}=1.0\). All couplings are ferromagnetic and have unit magnitude. This results in an energy landscape with a deep local minimum where both clusters are aligned to their respective biases, but a slightly lower global minimum when the two clusters are aligned together, sacrificing the benefit of bias-alignment for the satisfaction of the intercluster couplings. For all WSC runs, the spins of the lattice are initialized in the local minimum.
+
+The second class of Hamiltonians are nearest-neighbour square spin glasses (SG; see Figure 1b). Couplings are periodic (i.e., the model is defined on a torus), and drawn from a normal distribution with standard deviation 1.0. All biases are zero. Hamiltonian instances are generated as needed during training. To evaluate our method and compare against the simulated annealing standard, we must have a testing set of instances for which we know the true ground state. For each lattice size investigated (\(\sqrt{L}=[4,6,8,10,12,14,16]\)) we generate \(N_{\text{test}}=100\) unique instances and obtain the true ground state energy for each instance using the branch-and-cut method [52] through the Spin Glass Server [53].
+
+## VII Results
+
+### Weak-strong clusters model
+
+We demonstrate the use of RL on the WSC model shown in Figure 1a. RL is able to learn a simple temperature schedule that anneals the WSC model to the ground state in 100% of episodes, regardless of the temperature in which the system is initialized. In Figure 3b, we compare the RL policy to standard simulated annealing (i.e., Metropolis-Hastings) with several linear inverse temperature schedules (i.e., constant cooling rates).
+
+When the system is initially hot (small \(\beta\)), both RL and SA are capable of reaching the ground state with 100% success as there exists sufficient thermal energy to escape the local minimum. In Figure 3c, we plot an example schedule. The RL policy (red) increases the temperature slightly at first, but then begins to cool the system almost immediately. An abrupt decrease in the Metropolis acceptance rate is observed (Figure 3e). The blue dashed line in Figure 3c represents the average schedule of the RL policy over 100 independent anneals. The standard deviation is shaded. It is apparent that the schedule is quite consistent between runs at a given starting temperature, with some slight variation in the rate of cooling.
+
+When the system is initially cold (large \(\beta\)), there exists insufficient thermal energy to overcome the energy barrier between the local and global minima, and SA fails with a constant cooling rate. The RL policy, however, is able to identify, through observation, that the temperature is too low and can rapidly decrease \(\beta\) initially, heating the system to provide sufficient thermal energy to avoid the local minimum. In Figure 3f, we see an increase in the Metropolis acceptance ratio, followed by a decrease, qualitatively consistent with the human-devised heuristic schedules that have been traditionally suggested [11, 10, 1]. In Figure 3d, we plot an example schedule. Initially, the RL algorithm increases the temperature to provide thermal energy to escape the minimum, then begins the process of cooling. Similar to Figure 3c, the broadness of the variance of the policies is greatest in the cooling phase, with some instances being cooled more rapidly than others. The RL agent does not have access to the current temperature directly, and bases its policy solely on the spins. The orthogonal unit-width convolutions provide a mechanism for statistics over spins and replicas, and the LSTM module provides a mechanism to capture the time-dependent dynamics of the system.
+
+### Spin-glass model
+
+We now investigate the performance of the RL algorithm in learning a general policy for an entire class of Hamiltonians, investigating whether the RL algorithm can learn to generalize its learning to accommodate a theoretically infinite set of Hamiltonians of a specific class.
+
+Figure 4: An RL policy learns to anneal spin-glass models. An example (\(L=4^{2}\)) lattice is shown in (a). In (b) and (c), we plot the acceptance ratios over time for three episodes for each of the \(L=8^{2}\) and \(L=16^{2}\) lattices. In (d), we compare the scaling of the RL policy with respect to system size and compare it to a simple linear simulated annealing schedule. We plot the \(n_{99}\) value (the number of anneals required to be 99% certain of observing the ground state) as a function of system size for both the RL and the best linear simulated annealing schedule we observed. For all system sizes investigated, the learned RL policy is able to reach the ground state in significantly fewer runs. Additionally, we plot the destructive observation results, which in most cases, still rival or beat the linear schedules. We note that the destructive observation requires far more Monte Carlo steps per episode to simulate the destructive measurements; this plot should not be interpreted as a comparison of run time with regard to the destructive observation result. In (e) through (k), we plot an example inverse temperature schedule as a solid line, as well as the average inverse temperature schedule (for all testing episodes) as a dashed line, with the shaded region denoting the standard deviation.
+
+Figure 3: An RL policy learns to anneal the WSC model (shown in (a)). (b) We plot the performance of various linear (with respect to \(\beta\)) simulated annealing schedules, cooling from \(\beta_{i}\) to \(\beta_{f}\), as well as the performance of the RL policy for a variety of starting temperatures. When the initial inverse temperature is sufficiently small, both the RL and SA algorithms achieve 100% success (i.e., every episode reaches the ground state). When the system is initialized with a large \(\beta_{i}\), there is insufficient thermal energy for SA to overcome the energy barrier and reach the ground state, and consequently a very low success probability. A single RL policy achieves almost perfect success across all initial temperatures. In (c) and (d), we plot the RL inverse temperature schedule in red for episodes initialized with respective low and high inverse temperatures. In blue, we show the average RL policy for the specific starting temperature. The RL algorithm can identify a cold initialization from observation, and increases the temperature before then decreasing it (as shown in (d)). In (e) and (f), we plot the Metropolis acceptance ratio for two episodes, initialized at two extreme temperatures (e) low \(\beta_{i}\), and (f) high \(\beta_{i}\).
+
+Furthermore, we investigate how RL performs with various lattice sizes, and compare the trained RL model to a linear (with respect to \(\beta\)) classic simulated annealing schedule. The results of this investigation are shown in Figure 4.
+
+In all cases, the RL schedule obtains a better (lower) \(n_{99}\) value, meaning far fewer episodes are required for us to be confident that the ground state has been observed. Furthermore, the \(n_{99}\) value exhibits much better scaling with respect to the system size (i.e., the number of optimization variables). In Figure 4e-k, we plot some of the schedules that the RL algorithm produces. In many cases, we see initial heating, followed by cooling, although in the case of the \(L=16^{2}\) model (Figure 4k) we see much more complex, but still successful, behaviour. In all cases, the variance of the policies with respect to time (shown as the shaded regions in Figure 4e-k), indicate the agent is using information from the provided state to make decisions, and not just basing its decisions on the elapsed time using the internal state of the LSTM module. If schedules were based purely on some internal representation of time, there would be no variance between episodes.
+
+### Destructive observation
+
+A key element of the nature of quantum systems is the collapse of the wavefunction when a measurement of the quantum state is made. When dealing with quantum systems, one must make control decisions based on quantum states that have evolved through an identical policy but have never before been measured. We model this restriction on quantum measurements by allowing any replica observed in the anneal to be consumed as training data for the RL algorithm only once. We simulate this behaviour by keeping track of the policy decisions (the changes in inverse temperature) in an action buffer as we play through each episode. When a set of \(N_{\text{reps}}\) replicas are measured, they are consumed and the system is reinitialized. The actions held in the buffer are replayed on the new replicas.
+
+In this situation, the agent cannot base its decision on any replica-specific temporal correlations between given measurements; this should not be a problem early in each episode, as the correlation time scale of a hot system is very short, and the system, even under nondestructive observation, would have evolved sufficiently, in the time window between steps, to be uncorrelated. However, as the system cools, the correlation time scale increases exponentially, and destructive observation prevents the agent from relying on temporal correlations of any given replica.
+
+We evaluate an agent trained in this "quantum-inspired" way and plot its performance alongside the nondestructive (i.e., classical) case in Figure 4d. In the case of destructive observation, the agent performs marginally less well than the nondestructive case, but still performs better than SA in most cases. As it is a more complicated task to make observations when the system is temporally uncorrelated, it is understandable that the performance would be inferior to the nondestructive case. Nonetheless, RL is capable of outperforming SA in both the destructive and nondestructive cases.
+
+The relative performance in terms of computational demand between destructive observation and SA alludes to an important future direction in the field of RL, especially when applied to physical systems where observation is destructive, costly, and altogether difficult. With destructive observations, \(N_{\text{steps}}\) systems must be initialized and then evolved together under the same policy. Each copy is consumed one by one, as observations are required for decision making, thus incurring an unavoidable \(N_{\text{steps}}^{2}/2\) penalty in the destructive case. In this sense, it is difficult to consider RL to be superior; prescheduled SA simply does not require observation. However, if the choice to observe were to be incorporated into the action set of the RL algorithm, the agent would choose when observation would be necessary.
+
+For example, in the systems presented in this work, the correlation time of the observations is initially small; the temperatures are high, and frequent observations are required to guide the system through phase space. As the system cools, however, the correlation time grows exponentially, and the observations become much more similar to each previous observation; in this case, it would be beneficial to forgo some expensive observations, as the system would not be evolving substantially. With such a scheme, RL stands a better chance at achieving greater performance.
+
+### Policy analysis
+
+To glean some understanding into what the RL agent is learning, we train an additional model on a well-understood Hamiltonian, the ferromagnetic Ising model of size \(16\times 16\). In this case, the temperatures are initialized randomly (as in the WSC model). This model is the extreme case of a spin glass, with all \(J_{ij}=1\). In Figure 5a, we display the density of states \(g(M,E)\) of the Ising model, plotted in phase space, with axes of magnetization per spin (\(M/L\)) and energy per spin (\(E/L\)). The density of states is greatest in the high-entropy \(M=E=0\) region, and lowest in the low-entropy "corners". We show the spin configurations at the three corners ("checkerboard", spin-up, and spin-down) for clarity. The density of states is obtained numerically using Wang-Landau sampling [54]. Magnetization and energy combinations outside of the dashed "triangle" are impossible.
+
+In Figure 5b, we plot a histogram of the average value function \(V(s_{t})\) on the phase plane, as well as three trajectories. Note that since each observation \(s_{t}\) is composed of \(N_{\text{reps}}\) replicas, we count each observation as \(N_{\text{reps}}\) separate points on the phase plot when computing the histogram, each with an identical contribution of \(V(s_{t})\) to the average. As expected, the learned value function trends higher toward the two global energy minima. The lowest values are present in the initialization region (the high-energy band along the top). We expand two regions of interest in Figure 5c-d. In Figure 5d, we can see that the global minimum is assigned the highest value; this is justifiable in that if the agent reaches this point, it is likely to remain here and reap a high reward so long as the agent keeps the temperature low for the remainder of the episode.
+
+In Figure 5c, we identify four noteworthy energy-magnetization combinations, using asterisks. These four energy-magnetization combinations have identical energies, with increasing magnetization, and correspond to banded spin structures of decreasing width (four example spin configurations are shown). The agent learns to assign a higher value to the higher-magnetization structures, even though the energy, which is the true measure of "success", is identical. This is because the higher-magnetization bands are closer to the right-most global minimum in action space, that is, the agent can traverse from the small-band configuration to the ground state in fewer spin flips than if traversing from the wide-band configurations.
+
+In Figure 5e, we plot a histogram of the average action taken at each point in phase space. The upper high-energy band exhibits more randomness in the actions chosen, as this is the region in which the system lands upon initialization. When initialized, the temperature is at a randomly drawn value, and sometimes the agent must first heat the system to escape a local minimum before then cooling, and thus the first action is, on average, of very low magnitude. As the agent progresses toward the minimum, the agent becomes more aggressive in cooling the system, thereby thermally trapping itself in lower energy states.
+
+## VIII Conclusion
+
+In this work, we show that reinforcement learning is a viable method for learning dynamic control schemes for the task of simulated annealing (SA). We show that, on a simple spin model, a reinforcement learning (RL) agent is capable of devising a temperature control scheme that can consistently escape a local minimum, and then anneal to the ground state. It arrives at a policy that generalizes to a range of initialization temperatures; in all cases, it learns to cool the system. However, if the initial temperature is too low, the RL agent learns to first increase the temperature to provide sufficient thermal energy to escape the local minimum.
+
+We then demonstrate that the RL agent is capable of learning a policy that can generalize to an entire class of Hamiltonians, and that the problem need not be restricted to a single set of couplings. By training multiple RL agents on increasing numbers of variables (increasing lattice sizes), we investigate the scaling of the RL algorithm and find that it outperforms a linear SA schedule both in absolute terms and in terms of its scaling.
+
+We analyze the value function that the agent learns and see that it attributes an intuitive representation of value to specific regions of phase space.
+
+We discuss the nature of RL in the physical sciences, specifically in situations where observing systems is de
+
+Figure 5: We train an agent on a special case of the spin-glass Hamiltonians: the \(16\times 16\) ferromagnetic Ising model where all couplings \(J_{ij}=1\). (a) We plot the density of states \(\log(g(M,E))\) for the \(16\times 16\) Ising model in the phase space of energy and magnetization, sampled numerically using the Wang–Landau algorithm [54], and indicate four of the novel high- and low-energy spin configurations on a grid. (b) For the trained model, we plot the average of the learned value function \(V(s_{t})\) for each possible energy–magnetization pair. Additionally, we plot the trajectories of the first replica for three episodes of annealing to demonstrate the path through phase space the algorithm learns to take. In (c) and (d), we enlarge two high-value regions of interest. In (e), we plot the average action taken at each point in phase space, as well as the same two trajectories plotted in (b).
+
+structive ("destructive observation") or costly (e.g., performing quantum computations where observations collapse the wavefunction, or conducting chemical analysis techniques that destroy a sample material). We demonstrate that our implementation of RL is capable of performing well in a destructive observation situation, albeit inefficiently. We propose that the future of physical RL (i.e., RL in the physical sciences) will be one of "controlled observation", where the algorithm can choose when an observation is necessary, minimizing the inherent costs incurred when observations are expensive, slow, or difficult.
+
+## IX Acknowledgements
+
+I.T. acknowledges support from NSERC. K.M. acknowledges support from Mitacs. The authors thank Marko Bucyk for reviewing and editing the manuscript.
+
+## References
+
+* Kirkpatrick _et al._ [1983]S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, Science **220**, 671 (1983).
+* Barahona [1982]F. Barahona, Journal of Physics A: Mathematical and General **15**, 3241 (1982).
+* Sherrington and Kirkpatrick [1975]D. Sherrington and S. Kirkpatrick, Physical Review Letters **35**, 1792 (1975).
+* Ising [1925]E. Ising, Zeitschrift fur Physik **31**, 253 (1925).
+* Onsager [1944]L. Onsager, Physical Review **65**, 117 (1944).
+* Ferdinand and Fisher [1969]A. E. Ferdinand and M. E. Fisher, Physical Review **185**, 832 (1969).
+* Lucas [2014]A. Lucas, Frontiers in Physics **2**, 1 (2014).
+* Hastings [1970]B. Y. W. K. Hastings,, 97 (1970).
+* Metropolis _et al._ [1953]N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, The Journal of Chemical Physics **21**, 1087 (1953).
+* Kirkpatrick [1984]S. Kirkpatrick, Journal of Statistical Physics **34**, 975 (1984).
+* van Laarhoven and Aarts [1987]P. J. M. van Laarhoven and E. H. L. Aarts, _Simulated Annealing: Theory and Applications_ (Springer Netherlands, Dordrecht, 1987).
+* Bounds [1987]D. G. Bounds, Nature **329**, 215 (1987).
+* Farhi _et al._ [2001]E. Farhi, J. Goldstone, S. Gutmann, J. Lapan, A. Lundgren, and D. Preda, Science **292**, 472 (2001), arXiv:0104129 [quant-ph].
+* Atomic, Molecular, and Optical Physics **86**, 1 (2012), arXiv:1207.1712.
+* Boixo _et al._ [2014]S. Boixo, T. F. Ronnow, S. V. Isakov, Z. Wang, D. Wecker, D. A. Lidar, J. M. Martinis, and M. Troyer, Nature Physics **10**, 218 (2014), arXiv:1304.4595.
+* Bian _et al._ [2014]Z. Bian, F. Chudak, R. Israel, B. Lackey, W. G. Macready, and A. Roy, Frontiers in Physics **2**, 1 (2014).
+* Venturelli _et al._ [2015]D. Venturelli, D. J. J. Marchand, and G. Rojo,, 1 (2015), arXiv:1506.08479.
+* Ray _et al._ [1989]P. Ray, B. K. Chakrabarti, and A. Chakrabarti, Physical Review B **39**, 11828 (1989).
+* Condensed Matter and Materials Physics **66**, 1 (2002).
+* Santoro [2002]G. E. Santoro, Science **295**, 2427 (2002), arXiv:0205280 [cond-mat].
+* Finnila _et al._ [1994]A. Finnila, M. Gomez, C. Sebenik, C. Stenson, and J. Doll, Chemical Physics Letters **219**, 343 (1994).
+* Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics **58**, 5355 (1998), arXiv:9804280 [cond-mat].
+* Condensed Matter and Materials Physics **82**, 1 (2010), arXiv:1004.1628.
+* Johnson _et al._ [2011]M. W. Johnson, M. H. Amin, S. Gildert, T. Lanting, F. Hamze, N. Dickson, R. Harris, A. J. Berkley, J. Johansson, P. Bunyk, E. M. Chapple, C. Enderud, J. P. Hilton, K. Karimi, E. Ladizinsky, N. Ladizinsky, T. Oh, I. Perminov, C. Rich, M. C. Thom, E. Tolkacheva, C. J. Truncik, S. Uchaikin, J. Wang, B. Wilson, and G. Rose, Nature **473**, 194 (2011).
+* McGeoch and Wang [2013]C. C. McGeoch and C. Wang, Proceedings of the ACM International Conference on Computing Frontiers, CF 2013 10.1145/2482767.2482797 (2013).
+* Ikeda _et al._ [2019]K. Ikeda, Y. Nakamura, and T. S. Humble, Scientific Reports **9**, 1 (2019), arXiv:1904.12139.
+* Dickson _et al._ [2013]N. G. Dickson, M. W. Johnson, M. H. Amin, R. Harris, F. Altomare, A. J. Berkley, P. Bunyk, J. Cai, E. M. Chapple, P. Chavez, F. Ciosta, T. Cirip, P. Debuen, M. Drewbrook, C. Enderud, S. Gildert, F. Hamze, J. P. Hilton, E. Hoskinson, K. Karimi, E. Ladizinsky, N. Ladizinsky, T. Lanting, T. Mahon, R. Neufeld, T. Oh, I. Perminov, C. Petroff, A. Przybysz, C. Rich, P. Spear, A. Tcaciuc, M. C. Thom, E. Tolkacheva, S. Uchaikin, J. Wang, A. B. Wilson, Z. Merali, and G. Rose, Nature Communications **4**, 1 (2013).
+* Okada _et al._ [2019]S. Okada, M. Ohzeki, and K. Tanaka, (2019), arXiv:1904.01522.
+* Statistical, Nonlinear, and Soft Matter Physics **71**, 1 (2005), arXiv:0502468 [cond-mat].
+* Tsukamoto _et al._ [2017]S. Tsukamoto, M. Takatsu, S. Matsubara, and H. Tamura, Fujitsu Scientific and Technical Journal **53**, 8 (2017).
+* Inagaki _et al._ [2016]T. Inagaki, Y. Haribara, K. Igarashi, T. Sonobe, S. Tamate, T. Honjo, A. Marandi, P. L. McMahon, T. Umeki, K. Enbutsu, O. Tadanaga, H. Takenouchi, K. Aihara, K. I. Kawarabayashi, K. Inoue, S. Utsunomiya, and H. Takesue, Science **354**, 603 (2016).
+* Farhi _et al._ [2014]E. Farhi, J. Goldstone, and S. Gutmann,, 1 (2014), arXiv:1411.4028.
+
+* Farhi _et al._ [2019]E. Farhi, J. Goldstone, S. Gutmann, and L. Zhou,, 1 (2019), arXiv:1910.08187.
+* Sutton and Barto [2018]R. Sutton and A. Barto, _Reinforcement Learning: An Introduction_, 2nd ed. (The MIT Press, 2018).
+* Berner _et al._ [2019]C. Berner, G. Brockman, B. Chan, V. Cheung, P. P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Jozefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. Ponde, D. O. Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang, (2019), arXiv:1901.08004.
+* Vinyals _et al._ [2019]O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wunsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, Nature 10.1038/s41586-019-1724-z (2019).
+* Mnih _et al._ [2013]V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, arXiv preprint arXiv: {...} }, 1 (2013), arXiv:1312.5602.
+* Silver _et al._ [2016]D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelva, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Nature **529**, 484 (2016).
+* Silver _et al._ [2017]D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Nature **550**, 354 (2017).
+* Silver _et al._ [2018]D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, Science **362**, 1140 (2018).
+* Agostinelli _et al._ [2019]F. Agostinelli, S. McAleer, A. Shmakov, and P. Baldi, Nature Machine Intelligence **1**, 356 (2019).
+* OpenAI _et al._ [2019]OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang,, 1 (2019), arXiv:1910.07113.
+* Schulman _et al._ [2017]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,, 1 (2017), arXiv:1707.06347.
+* Hill _et al._ [2018]A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, Stable baselines, [https://github.com/hill-a/stable-baselines](https://github.com/hill-a/stable-baselines) (2018).
+* Brockman _et al._ [2016]G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba,, 1 (2016), arXiv:1606.01540.
+* Hochreiter and Schmidhuber [1997]S. Hochreiter and J. Schmidhuber, Neural Computation **9**, 1735 (1997).
+* Schulman _et al._ [2015]J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel,, 1 (2015), arXiv:1506.02438.
+* Aramon _et al._ [2019]M. Aramon, G. Rosenberg, E. Valiante, T. Miyazawa, H. Tamura, and H. G. Katzgraber, Frontiers in Physics **7**, 10.3389/fphy.2019.00048 (2019), arXiv:1806.08815.
+* Bunyk _et al._ [2014]P. I. Bunyk, E. M. Hoskinson, M. W. Johnson, E. Tolkacheva, F. Altomare, A. J. Berkley, R. Harris, J. P. Hilton, T. Lanting, A. J. Przybysz, and J. Whittaker, IEEE Transactions on Applied Superconductivity **24**, 1 (2014), arXiv:1401.5504.
+* Liers _et al._ [2005]F. Liers, M. Junger, G. Reinelt, and G. Rinaldi, New Optimization Algorithms in Physics, 47 (2005).
+* Junger [2017]M. Junger, Spin Glass Server ([https://informatik.uni-koeln.de/spinglass/](https://informatik.uni-koeln.de/spinglass/)).
+* Wang and Landau [2001]F. Wang and D. P. Landau, Physical Review Letters **86**, 2050 (2001), arXiv:0011174 [cond-mat].

2003.00011v2.mmd

@@ -0,0 +1,317 @@
+# Finding the ground state of spin Hamiltonians with reinforcement learning
+
+Kyle Mills
+
+kyle.mills@1qbit.com 1QB Information Technologies (1QBit), Vancouver, British Columbia, Canada
+
+University of Ontario Institute of Technology, Oshawa, Ontario, Canada
+
+Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
+
+Pooya Ronagh
+
+pooya.ronagh@1qbit.com 1QB Information Technologies (1QBit), Vancouver, British Columbia, Canada
+
+Institute for Quantum Computing (IQC), Waterloo, Ontario, Canada
+
+Department of Physics and Astronomy, University of Waterloo, Ontario, Canada
+
+Isaac Tamblyn
+
+isaac.tamblyn@nrc.ca National Research Council Canada, Ottawa, Ontario, Canada
+
+University of Ottawa, Ottawa, Ontario, Canada
+
+Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
+
+November 5, 2021
+
+###### Abstract
+
+Reinforcement learning (RL) has become a proven method for optimizing a procedure for which success has been defined, but the specific actions needed to achieve it have not. Using a method we call "Controlled Online Optimization Learning" (COOL), we apply the so-called "black box" method of RL to simulated annealing (SA), demonstrating that an RL agent based on proximal policy optimization can, through experience alone, arrive at a temperature schedule that surpasses the performance of standard heuristic temperature schedules for two classes of Hamiltonians. When the system is initialized at a cool temperature, the RL agent learns to heat the system to "melt" it, and then slowly cool it in an effort to anneal to the ground state; if the system is initialized at a high temperature, the algorithm immediately cools the system. We investigate the performance of our RL-driven SA agent in generalizing to all Hamiltonians of a specific class; when trained on random Hamiltonians of nearest-neighbour spin glasses, the RL agent is able to control the SA process for other Hamiltonians, reaching the ground state with a higher probability than a simple linear annealing schedule. Furthermore, the scaling performance (with respect to system size) of the RL approach is far more favourable, achieving a performance improvement of almost two orders of magnitude on \(L=14^{2}\) systems. We demonstrate the robustness of the RL approach when the system operates in a "destructive observation" mode, an allusion to a quantum system where measurements destroy the state of the system. The success of the RL agent could have far-reaching impact, from classical optimization, to quantum annealing, to the simulation of physical systems.
+
+## I Introduction
+
+In metallurgy and materials science, the process of annealing is used to equilibrate the positions of atoms to obtain perfect low-energy crystals. Heat provides the energy necessary to break atomic bonds, and high-stress interfaces are eliminated by the migration of defects. By slowly cooling the metal to room temperature, the metal atoms become energetically locked in a lattice structure more favourable than the original structure. Metallurgists can tune the temperature schedule to arrive at final products that have desired characteristics, such as ductility and hardness. Annealing is a biased stochastic search for the ground state.
+
+An analogous _in silico_ technique, simulated annealing (SA) [1], can be used to find the ground state of spin-glass models, an NP-hard problem [2]. A spin glass is a graphical model consisting of binary spins \(\sigma_{i}\). The connections between spins are defined by the coupling constants \(J_{ij}\), and a linear term with coefficients \(h_{i}\) can apply a bias to individual spins. The Hamiltonian
+
+\[\mathcal{H}=-\sum_{i<j}J_{ij}\sigma_{i}\sigma_{j}-\sum_{i}h_{i}\sigma_{i}, \quad\sigma_{i}=\pm 1\]
+
+defines the energy of the microstates [3]. The choices of the quadratic coupling coefficients \(J_{ij}\) and the linear bias coefficients \(h_{i}\) effect the interesting dynamics of the model: \(J_{ij}\) can be randomly distributed according to a Gaussian distribution [3], encompass all \(i,j\) combinations for a fully connected Hamiltonian, or be limited to short-range (e.g., nearest-neighbour, \(\langle i,j\rangle\)) interactions, to name a few. For example, when the positive, unit-magnitude coupling is limited to nearest-neighbour pairs, the ubiquitous ferromagnetic Ising model [4] is recovered. Examples of the Hamiltonians we investigate in this work are presented in Figure 1 and discussed in further detail in Section IV.
+
+Finding the ground state of (i.e., "solving") such systems is interesting from the perspective of thermodynamics, as one can observe phenomena such as phase transitions [5; 6], but also practically useful as discrete optimization problems can be mapped to spin-glass models (e.g., the travelling salesperson problem or the knapsack problem) [7]. The Metropolis-Hastings algorithm [8; 9] can be used to simulate the spin glass at arbitrary temperature, \(T\); thus, it is used ubiquitously for SA. By beginning the simulation at a high temperature, one can slowly cool the system over time, providing sufficient thermal energy to escape local minima, and arrive at the ground state "solution" to the problem. The challenge is to find a temperature schedule that minimizes computational effort while still arriving at a satisfactory solution; if the temperature is reduced too rapidly, the system will become trapped in a local minimum, and reducing the temperature too slowly results in an unnecessary computational expense. Kirkpatrick et al. [1; 10] proposed starting at a temperature that results in an 80% acceptance ratio (i.e., 80% of Metropolis spin flips are accepted) and reducing the temperature geometrically. They also recommended monitoring the objective function and reducing the cooling rate if the objective value (e.g., the energy) drops too quickly. More-sophisticated adaptive temperature schedules have been investigated [11]; however, simple linear and reciprocal temperature schedules are commonly used in practice [12; 13]. We will refer to SA using a linear schedule as "classic SA" throughout this work. Nevertheless, in his 1987 paper, Bounds [14] said that "choosing an annealing schedule for practical purposes is still something of a black art".
+
+When framed in the advent of quantum computation and quantum control, establishing robust and dynamic scheduling of control parameters becomes even more relevant. For example, the same optimization problems that can be cast as classical spin glasses are also amenable to quantum annealing [15; 16; 17; 18; 19], exploiting, in lieu of thermal fluctuations, the phenomenon of quantum tunnelling [20; 21; 22] to escape local minima. Quantum annealing (QA) was proposed by Finnila et al. [23] and Kadowaki and Nishimori [24], and, in recent years, physical realizations of devices capable of performing QA (quantum annealers), have been developed [25; 26; 27; 28], and are being rapidly commercialized. As these technologies progress and become more commercially viable, practical applications [19; 29] will continue to be identified and resource scarcity will spur the already extant discussion of the efficient use of annealing hardware [30; 31].
+
+Nonetheless, there are still instances where the classical (SA) outperforms the quantum (QA) [32], and improving the former should not be undervalued. _In silico_ and hardware annealing solutions such as Fujitsu's FPGA-based Digital Annealer [33], NTT's laser-pumped coherent Ising machine (CIM) [34; 35; 36], and the quantum circuit model algorithm known as QAOA [37; 38] all demand the scheduling of control parameters, whether it is the temperature in the case of the Digital Annealer, or the power of the laser pump in the case of CIM. Heuristic methods based on trial-and-error experiments are commonly used to schedule these control parameters, and an automatic approach could expedite development, and improve the stability of such techniques.
+
+In this work, we demonstrate the use of a reinforcement learning (RL) method to learn the "black art" of SA temperature scheduling, and show that an RL agent is able to learn dynamic control parameter schedules for various problem Hamiltonians. The schedules that the RL agent produces are dynamic and reactive, adjusting to the current observations of the system to reach the ground state quickly and consistently without _a priori_ knowledge of a given Hamiltonian. We believe that RL will be important for quantum information processing, especially for hardware- and software-based control.
+
+Figure 1: Two classes of Hamiltonian problems are depicted. (a) The weak-strong clusters (WSC) model comprises two bipartite clusters. The left cluster is biased upward (\(h_{i}>0\)); the right cluster is biased downward (\(h_{i}<0\)). All couplings are equal and of unit magnitude. The two clusters are coupled via the eight central nodes. This model exhibits a deep local minimum very close in energy to the model’s global minimum. When initialized in the local minimum, the RL agent is able to learn schemes to escape the local minimum and arrive at the global minimum, without any explicit knowledge of the Hamiltonian. (b) Here we present an example spin-glass model. The nodes are coupled to nearest neighbours with random Gaussian-distributed coupling coefficients. The nodes are unbiased (\(h_{i}=0\)), and the couplings are changed at each instantiation of the model. The RL algorithm is able to learn a dynamic temperature schedule by observing the system throughout the annealing process, without explicit knowledge of the form of the Hamiltonian, and the learned policy can be applied to all instances of randomly generated couplings. We demonstrate this on variably sized spin glasses and investigate the scaling with respect to a classic linear SA schedule. In (c), we show snapshots of a sample progression of a configuration undergoing SA under the ferromagnetic Ising model Hamiltonian and a constant cooling schedule. The terminal state, all spins-up, is the ground state; this anneal would be considered successful.
+
+## II The environment and architecture
+
+### Reinforcement learning
+
+Reinforcement learning is a branch of dynamic programming whereby an agent, residing in state \(s_{t}\) at time \(t\), learns to take an action \(a_{t}\) that maximizes a cumulative reward signal \(R\) by dynamically interacting with an environment [39]. Through the training process, the agent arrives at a policy \(\pi\) that depends on some observation (or "state") of the system, \(s\). In recent years, neural networks have taken over as the _de facto_ function approximator for the policy. Deep reinforcement learning has seen unprecedented success, achieving superhuman performance in a variety of video games [40, 41, 42, 43], board games [44, 45, 46], and other puzzles [47, 48]. While many reinforcement learning algorithms exist, we have chosen to use proximal policy optimization (PPO) [49], implemented within Stable Baselines [50] for its competitive performance on problems with continuous action spaces.
+
+### The environment
+
+We developed an OpenAI gym [51] environment which serves as the interface to the "game" of simulated annealing. Let us now define some terminology and parameters important to SA. For a given Hamiltonian, defining the interactions of \(L\) spins, we create \(N_{\text{reps}}\) randomly initialized replicas (unless otherwise specified). The initial spins of each replica are drawn from a Bernoulli distribution with probability of a spin-up being randomly drawn from a uniform distribution. These independent replicas are annealed in parallel. The replicas follow an identical temperature schedule with their uncoupled nature providing a mechanism for statistics of the system to be represented through an ensemble of measurements. In the context of the Metropolis-Hastings framework, we define one "sweep" to be \(L\) proposed random spin flips (per replica), and one "step" to be \(N_{\text{sweeps}}\). After every step, the environment returns an observation of the current state \(s_{t}\) of the system, an \(N_{\text{reps}}\times L\) array consisting of the binary spin values present. This observation can be used to make an informed decision of the action \(a_{t}\) that should be taken. The action, a single scalar value, corresponds to the total inverse temperature change \(\Delta\beta\) (where \(\beta=1/T\)) that should be carried out over the subsequent step. The choice of action is provided to the environment, and the process repeats until \(N_{\text{steps}}\) steps have been taken, comprising one full anneal, or "episode" in the language of RL. If the chosen action would result in the temperature becoming negative, no change is made to the temperature and the system continues to evolve under the previous temperature.
+
+In our investigations, we choose \(N_{\text{steps}}=40\) and \(N_{\text{sweeps}}=100\) resulting, in 4000 sweeps per episode. These values define the maximum size of system we can compare to classic SA. This number of sweeps is sufficient for a linear schedule to attain measurable success on all but the largest system size we investigate.
+
+### Observations
+
+For the classical version of the problem, an observation consists of the explicit spins of an ensemble of replicas. In the case of an unknown Hamiltonian, the ensemble measurement is important as the instantaneous state of a single replica does not provide sufficient information about the current temperature of the system. Provid
+
+Figure 2: A neural network is used to learn the control parameters for several SA experiments. By observing a lattice of spins, the neural network can learn to control the temperature of the system in a dynamic fashion, annealing the system to the ground state. The spins at time \(t\) form the state \(s_{t}\) fed into the network. Two concurrent convolutional layers extract features from the state. These features are combined with a dense layer and fed into a recurrent module (an LSTM module) capable of capturing temporal characteristics. The LSTM module output is reduced to two parameters used to form the policy distribution \(\pi_{\theta}(a_{t}\mid s_{t})\) as well as to approximate the value function \(V(s_{t})\) used for the generalized advantage estimate.
+
+ing the agent with multiple replicas allows it to compute statistics and have the possibility of inferring the temperature. For example, if there is considerable variation among replicas, then the system is likely hot, whereas if most replicas look the same, the system is probably cool.
+
+When discussing a quantum system, where the spins represent qubits, direct mid-anneal measurement of the system is not possible as measurement causes a collapse of the wavefunction. To address this, we discuss experiments conducted in a "destructive observation" environment, where measurement of the spins is treated as a "one-time" opportunity for inclusion in RL training data. The subsequent observation is then based on a different set of replicas that have evolved through the same schedule, but from different initializations.
+
+When running the classic SA baselines, to keep comparison fair, each episode consists of \(N_{\text{reps}}\) replicas as in the RL case. If even one replica reaches the ground state, the episode is considered a success.
+
+### Reinforcement learning algorithm
+
+Through the framework of reinforcement learning, we wish to produce a policy function \(\pi_{\theta}(a_{t}\mid s_{t})\) that takes the observed binary spin state \(s_{t}\in\{-1,1\}^{N_{\text{rep}}\times L}\) and produces an action \(a_{t}\) corresponding to the optimal change in the inverse temperature.
+
+We now briefly introduce PPO [49]. First we define our policy \(\pi_{\theta}(a_{t}\mid s_{t})\) as the likelihood that the agent will take action \(a_{t}\) while in state \(s_{t}\); through training, the desire is that the best choice of action will become the most probable. To choose an action, this distribution can be sampled. We will use a neural network, parameterized by weights \(\theta\) to represent the policy by assuming that \(\pi_{\theta}(a_{t}\mid s_{t})\) is a normal distribution and interpreting the output nodes of the neural network as the mean, \(\mu\), and variance, \(\sigma^{2}\).
+
+We define a function \(Q_{\pi_{\theta}}(s_{t},a_{t})\) as the expected future discounted reward if the agent takes action \(a_{t}\) at time \(t\) and then follows policy \(\pi_{\theta}\) for the remainder of the episode. We additionally define a value function \(V_{\pi_{\theta}}(s_{t})\) as the expected future discounted reward starting from state \(s_{t}\) and following the current policy \(\pi_{\theta}\) until the end of the episode. We introduce the concept of _advantage_, \(\hat{A}_{t}(s_{t},a_{t})\), as the difference between these two quantities. \(Q_{\pi_{\theta}}\) and \(V_{\pi_{\theta}}\) are not known and must be approximated. We assume the features necessary to represent \(\pi\) are generally similar to the features necessary to estimate the value function, and thus we can use the same neural network to predict the value function by merely having it output a third quantity.
+
+\(\hat{A_{t}}\) is effectively an estimate of how much better the agent did in choosing action \(a_{t}\), compared to what was expected. We construct the typical policy gradient cost function by coupling the advantage of a state-action pair with the probability of the action being taken,
+
+\[L^{\text{PG}}(\theta)=\hat{\mathbb{E}}_{t}\left[\log\pi_{\theta}(a_{t}\mid s_ {t})\hat{A}_{t}\right],\]
+
+which we want to maximize by modifying the weights \(\theta\) through the training process. It is, however, more efficient to maximize the improvement ratio \(r_{t}\) of the current policy over a policy from a previous iteration \(\pi_{\theta_{\text{old}}}\)[52; 53]:
+
+\[L^{\text{TRPO}}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})}\hat{A}_{t}\right] \equiv\hat{\mathbb{E}}_{t}\left[r_{t}(\theta)\hat{A}_{t}\right].\]
+
+Note, however, that maximizing this quantity can be trivially achieved by making the new policy drastically different from the old policy, which is not the desired behaviour. The PPO algorithm [49] deals with this by clipping the improvement and taking the minimum
+
+\[L^{\text{CLIP}}(\theta)=\hat{\mathbb{E}}_{t}\left[\min(r_{t}(\theta)\hat{A}_{ t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\right].\]
+
+To train the value function estimator, a squared error is used, that is,
+
+\[L^{\text{VF}}(\theta)=\hat{\mathbb{E}}_{t}[(V_{\pi_{\theta}}(s_{t})-V_{t}^{ \text{targ}})^{2}],\]
+
+and to encourage exploration, an entropic regularization functional \(S\) is used. This all amounts to a three-term cost function
+
+\[L^{\text{PPO}}(\theta)=\hat{\mathbb{E}}_{t}\left[L^{\text{CLIP}}(\theta)-c_{1 }L^{\text{VF}}(\theta)+c_{2}S[\pi_{\theta}](s_{t})\right],\]
+
+where \(c_{1}\) and \(c_{2}\) are hyperparameters.
+
+### Policy network architecture
+
+The neural network is composed of two parts: a convolutional feature extractor, and a recurrent network to capture the temporal characteristics of the problem. The feature extractor comprises two parallel two-dimensional convolutional layers. The first convolutional layer has \(N_{k_{r}}\) kernels of size \(1\times L\), and aggregates along the replicas dimension, enabling the collection of spin-wise statistics across the replicas. The second convolutional layer has \(N_{k_{z}}\) kernels of size \(N_{\text{reps}}\times 1\) and slides along the spin dimension, enabling the aggregation of replica-wise statistics across the spins. The outputs of these layers are flattened, concatenated, and fed into a dense layer of size \(N_{d}\) hidden nodes. This operates as a latent space encoding for input to a recurrent neural network (a long short-term memory, or LSTM, module [54]), used to capture the sequential nature of our application. The latent output of the LSTM module is of size \(N_{L}\). For simplicity, we set \(N_{k_{r}}=N_{k_{s}}=N_{d}=N_{L}=64\). All activation functions are hyperbolic tangent (tanh) activations. Since \(a_{t}\) can assume a continuum of real values, this task is referred to as having a continuous action space, and thus standard practice is for the network to output two values corresponding to the first and second moments of a normal distribution, which can be sampled to produce predictions.
+
+### Reward
+
+At the core of RL is the concept of reward engineering, that is, developing a reward scheme to inject a notion of success into the system. As we only care about reaching the ground state by the end of a given episode, we use a sparse reward scheme, with a reward of zero for every time step before the terminal step, and a reward equal to the negative of the minimum energy as the reward for the terminal step, that is,
+
+\[R_{t}=\begin{cases}0,&t<N_{\text{steps}}\\ -\min_{k}\mathcal{H}(\phi_{k}(s_{t})),&t=N_{\text{steps}}\end{cases}, \tag{1}\]
+
+where \(k\in[1,N_{\text{reps}}]\), and
+
+\[\phi_{k}(s_{t})\in\{-1,1\}^{1\times L}\]
+
+is an indexing function that returns the binary spin values for the \(k\)-th replica of state \(s_{t}\). This reward function is agnostic to system size; as the system size increases, the correlation time will also increase, and additional sweeps may be required between actions, but the reward function remains applicable. Furthermore, with this reward scheme, we encourage the agent to arrive at the lowest possible energy by the time the episode terminates, without regard to what it does in the interim. In searching for the ground state, the end justifies the means.
+
+### Hyperparameters
+
+When optimizing the neural network, we use a PPO discount factor of \(\gamma=0.99\), eight episodes between weight updates, a value function coefficient of \(c_{1}=0.5\), an entropy coefficient of \(c_{2}=0.001\), a clip range of \(\epsilon=0.05\), a learning rate of \(\alpha=1\times 10^{-6}\), and a single minibatch per update. Each agent is trained over the course of \(25,000\) episodes (anneals), with \(N_{\text{steps}}=40\) steps per episode, and with \(N_{\text{sweeps}}=100\) sweeps separating each observation. We used \(N_{\text{reps}}=64\) replicas for each observation.
+
+## III Evaluation
+
+Whereas the RL policy can be made deterministic, meaning a given state always produces the same action, the underlying Metropolis algorithm is stochastic; thus, we must statistically define the metric for success. We borrow this evaluation scheme from Aramon et al. [55]. Each RL episode will either result in "success" or "failure". Let us define the "time to solution" as
+
+\[T_{s}=\tau n_{99}\,, \tag{2}\]
+
+that is, the number of episodes that must be run to be \(99\%\) sure the ground state has been observed at least one time (\(n_{99}\)), multiplied by the time \(\tau\) taken for one episode. As \(\tau\) depends specifically on the hardware used, and the efficiency of software implementations, we will focus on \(n_{99}\) alone as the metric we desire to minimize.
+
+Let us also define \(X_{i}\) as the binary outcome of the \(i\)-th episode, with \(X_{i}=1\) (\(0\)) if at least one (none) of the \(N_{\text{reps}}\) replicas are observed to be in the ground state at episode termination. The quantity \(Y\equiv\sum_{i=1}^{n}X_{i}\) is the number of successful episodes after a total of \(n\) episodes, and \(p\equiv P(X_{i}=1)\) denotes the probability that an anneal \(i\) will be successful. Thus the probability of exactly \(k\) out of \(n\) episodes succeeding is given by the probability mass function of the binomial distribution
+
+\[P(Y=k\mid n,p)=\begin{pmatrix}n\\ k\end{pmatrix}p^{k}(1-p)^{n-k}. \tag{3}\]
+
+To compute the time to solution, our quantity of interest is the number of episodes \(n_{99}\) where \(P=0.99\), that is,
+
+\[P(Y\geq 1\mid n_{99},p)=0.99.\]
+
+From this and (3), it can be shown that
+
+\[n_{99}=\frac{\log{(1-0.99)}}{\log{(1-p)}}.\]
+
+In the work of Aramon et al. [55], \(p\) is estimated using Bayesian inference due to their large system sizes sometimes resulting in zero successes, precluding the direct calculation of \(p\). In our case, to evaluate a policy, we perform \(100\) runs for each of \(100\) instances and compute \(p\) directly from the ratio of successful to total episodes, that is, \(p=\bar{X}\).
+
+## IV Hamiltonians
+
+We present an analysis of two classes of Hamiltonians. The first, which we call the weak-strong clusters model (WSC; see Figure 1a), is an \(L=16\) bipartite graph with two fully connected clusters, inspired by the "Chimera" structure used in D-Wave Systems' quantum annealing hardware [56]. In our case, one cluster is negatively biased with \(h_{i}=-0.44\) and the other positively biased with \(h_{i}=1.0\). All couplings are ferromagnetic and have unit magnitude. This results in an energy landscape with a deep local minimum where both clusters are aligned to their respective biases, but a slightly lower global minimum when the two clusters are aligned together, sacrificing the benefit of bias-alignment for the satisfaction of the intercluster couplings. For all WSC runs, the spins of the lattice are initialized in the local minimum.
+
+The second class of Hamiltonians are nearest-neighbour square spin glasses (SG; see Figure 1b). Couplings are periodic (i.e., the model is defined on a torus), and drawn from a normal distribution with standard deviation \(1.0\). All biases are zero. Hamiltonian instances are generated as needed during training. To evaluate our method and compare against classic SA, we must have a testing set of instances for which we know the true ground state. For each lattice size investigated (\(\sqrt{L}=[4,6,8,10,12,14,16]\)) we generate \(N_{\text{test}}=100\) unique instances and obtain the true ground state energy for each instance using the branch-and-cut method [57] through the Spin Glass Server [58].
+
+## V Results
+
+### Weak-strong clusters model
+
+We demonstrate the use of RL on the WSC model shown in Figure 1a. RL is able to learn a simple temperature schedule that anneals the WSC model to the ground state in 100% of episodes, regardless of the temperature in which the system is initialized. In Figure 3b, we compare the RL policy to classic SA schedules with several constant cooling rates.
+
+When the system is initially hot (small \(\beta\)), both RL and classic SA are capable of reaching the ground state with 100% success as there exists sufficient thermal energy to escape the local minimum. In Figure 3c, we plot an example schedule. The RL policy (red) increases the temperature slightly at first, but then begins to cool the system almost immediately. An abrupt decrease in the Metropolis acceptance rate is observed (Figure 3e). The blue dashed line in Figure 3c represents the average schedule of the RL policy over 100 independent anneals. The standard deviation is shaded. It is apparent that the schedule is quite consistent between runs at a given starting temperature, with some slight variation in the rate of cooling.
+
+When the system is initially cold (large \(\beta\)), there exists insufficient thermal energy to overcome the energy barrier between the local and global minima, and SA fails with a constant cooling rate. The RL policy, however, is able to identify, through observation, that the temperature is too low and can rapidly decrease \(\beta\) initially, heating the system to provide sufficient thermal energy to avoid the local minimum. In Figure 3f, we see an increase in the Metropolis acceptance ratio, followed by a decrease, qualitatively consistent with the human-devised heuristic schedules that have been traditionally suggested [1, 10, 11]. In Figure 3d, we plot an example schedule. Initially, the RL algorithm increases the temperature to provide thermal energy to escape the minimum, then begins the process of cooling. Similar to Figure 3c, the broadness of the variance of the policies is greatest in the cooling phase, with some instances being cooled more rapidly than others. The RL agent does not have access to the current temperature directly, and bases its policy solely on the spins. The orthogonal unit-width convolutions provide a mechanism for statistics over spins and replicas, and the LSTM module provides a mechanism to capture the time-dependent dynamics of the system.
+
+### Spin-glass model
+
+We now investigate the performance of the RL algorithm in learning a general policy for an entire class of Hamiltonians, investigating whether the RL algorithm can learn to generalize its learning to accommodate a theoretically infinite set of Hamiltonians of a specific class. Furthermore, we investigate how RL performs with various lattice sizes, and compare the trained RL model to a linear (with respect to \(\beta\)) classic SA schedule such as the ones used [12, 13]. The results of this investigation are shown in Figure 4.
+
+In all cases, the RL schedule obtains a better (lower) \(n_{99}\) value, meaning far fewer episodes are required for us to be confident that the ground state has been observed. Furthermore, the \(n_{99}\) value exhibits much better scaling with respect to the system size (i.e., the number of optimization variables). In Figure 4e-k, we plot some of the schedules that the RL algorithm produces. In many cases, we see initial heating, followed by cooling, although in the case of the larger models (Figure 4i-k) we see much more complex, but still successful, behaviour. In all cases, the variance of the policies with respect to time (shown as the shaded regions in Figure 4e-k), indicate the agent is using information from the provided state to make decisions, and not just basing its decisions on the elapsed time using the internal state of the LSTM module. If schedules were based purely on some internal representation of time, there would be no variance between episodes.
+
+### Comparing easy and difficult instances
+
+The learned strategy of the RL agent is relatively simple in concept: increase the temperature to a sufficiently high value and then use the remaining time to cool the system as seen in the average policies in Figure 4e-k. In this section we demonstrate the degree to which the performance improvement can be attributed to the ability of the RL agent to base its decisions upon the various dynamics in the system.
+
+We divide the instances in the \(10\times 10\) test set into two subsets, which we label "easy" and "difficult", based on the success of the classic SA baseline. This results in 14 difficult instances in which classic SA succeeds in only 3% of anneals, and 86 easy instances in which classic SA succeeds in more than 3% of anneals.
+
+Figure 4: An RL policy learns to anneal spin-glass models. An example (\(L=4^{2}\)) lattice is shown in (a). In (b) and (c), we plot the acceptance ratios over time for three episodes for each of the \(L=8^{2}\) and \(L=16^{2}\) lattices. In (d), we compare the scaling of the RL policy with respect to system size and compare it to classic SA. We plot the \(n_{99}\) value (the number of anneals required to be 99% certain of observing the ground state; in the case of 100% success, \(n_{99}\) is undefined and plotted as zero) as a function of system size for both the RL and the best linear simulated annealing schedule we observed. The 95% confidence interval is shown as a shaded region. For all system sizes investigated, the learned RL policy is able to reach the ground state in significantly fewer runs. Additionally, we plot the destructive observation results, which also outperform the linear schedules. We note that the destructive observation requires far more Monte Carlo steps per episode to simulate the destructive measurements; this plot should not be interpreted as a comparison of run time with regard to the destructive observation result. In (e) through (k), we plot an example inverse temperature schedule as a solid line, as well as the average inverse temperature schedule (for all testing episodes) as a dashed line, with the shaded region denoting the standard deviation. In this work, we use \(N_{\text{steps}}=40\) episode steps.
+
+Figure 3: An RL policy learns to anneal the WSC model (shown in (a)). (b) We plot the performance of various classic SA schedules, cooling linearly from \(\beta_{i}\) to \(\beta_{f}\), as well as the performance of the RL policy for a variety of starting temperatures. When the initial inverse temperature is sufficiently small, both the RL and classic SA algorithms achieve 100% success (i.e., every episode reaches the ground state). When the system is initialized with a large \(\beta_{i}\), there is insufficient thermal energy for classic SA to overcome the energy barrier and reach the ground state, and consequently a very low success probability. A single RL policy achieves almost perfect success across all initial temperatures. In (c) and (d), we plot the RL inverse temperature schedule in red for episodes initialized with respective low and high inverse temperatures. In blue, we show the average RL policy for the specific starting temperature. The RL algorithm can identify a cold initialization from observation, and increases the temperature before then decreasing it (as shown in (d)). In (e) and (f), we plot the Metropolis acceptance ratio for two episodes, initialized at two extreme temperatures (e) low \(\beta_{i}\), and (f) high \(\beta_{i}\). In this work, we use \(N_{\text{steps}}=40\) episode steps.
+
+We compare three temperature scheduling methods on 100 episodes of each instance in both of these subsets: i) classic (linear) SA; ii) the RL agent; and iii) an RL agent (not yet discussed) that does not include a recurrent LSTM module. As shown in Figure 5a, linearly scheduled classic SA solves the easy instances in 19% of anneals, whereas the reinforcement learning agent manages to solve the same instances with a 53% success probability. With the difficult instances, the difference is more extreme; classic SA manages only 1% success, whereas RL performs substantially better with 29% success.
+
+A variant of the agent without an LSTM module performs more poorly, but still better than classic SA. This agent is simply provided with a floating point representation of the episode step concatenated to the state vector, but without a recurrent network, it has no mechanism to capture the time dependence (history) of the problem. It therefore can only use the current observation in making decisions, and evidently does so more poorly than the agent with access to an LSTM module. For our formulation of the environment, an LSTM module is theoretically important to achieve a well-defined Markov decision process.
+
+In Figure 5b, we plot the average action taken, and in Figure 5c, we plot the average inverse temperature of the system at each step in the test episodes driven by the RL agent, averaged over the easy and difficult instances separately. There is no notable difference in the average schedules of the two subsets. This fact, combined with the considerable magnitude of the standard deviation (plotted as a filled region for difficult instances and vertical bars for easy ones) suggests that the RL agent is adaptive to the specific instantiation of each Hamiltonian. Some of these dynamics can be seen in the successful schedules randomly selected for plotting in Figure 5f.
+
+We then take the average schedules plotted in Figure 5b-c and use them as if they were RL-designed general heuristic schedules, removing the necessity to conduct observations during the training procedure. Both the difficult and easy average schedules perform very
+
+Figure 5: We separate the \(10\times 10\) spin glass instances in the test set into two subsets (easy and difficult), depending on the success of classic SA in finding their ground states. In (a) we plot the performance of three different temperature scheduling approaches on these subsets. RL exhibits superior performance over classic SA in both subsets; however, it demonstrates dramatic superiority in the case of the difficult instances. RL without an LSTM module still performs better than classic SA; it can still dynamically modify the schedule and is not constrained to a constant temperature change at each step, so is more akin to a traditional heuristic temperature scheduling approach. In (b) and (c) we plot the average RL actions and schedule, respectively, for both the difficult and easy instance subsets. The standard deviation of the policies are plotted with error bars (easy instances) and shaded regions (difficult instances). The average difficult policy is very similar to the average easy policy, both having a large standard deviation, suggesting a high degree of specificity of the policy to the given episode. We can see this in (f), where we plot several successful schedules; each schedule is quite different from the others, but each results in a successful episode. In (d), we show the performance when we apply the average actions presented in (b) as a static policy. The average policies perform even more poorly than classic SA. This is further evidence that the RL agent’s ability to observe the system is crucial to its high performance. One might object to the method used to split the instances into the difficult and easy subsets; we have explicitly chosen to split the subsets at a boundary that makes classic SA perform poorly on the difficult instances. Therefore in (e), we consider a difficult (easy) instance as one that the RL agent performs poorly (well) on, and the story remains unchanged.
+
+poorly on both the difficult and easy subsets, succeeding in less than 10% of episodes. This is strong evidence of the specificity of the RL agent's actions to the particular dynamics of each episode and refutes the hypothesis that a single, average policy, even if trained by RL, is a good case for generic instances.
+
+We repeat the previous analysis with subsets based on the performance of the RL agent, arriving at identical conclusions (Figure 5e).
+
+### Destructive observation
+
+A key element of the nature of quantum systems is the collapse of the wavefunction when a measurement of the quantum state is made. When dealing with quantum systems, one must make control decisions based on quantum states that have evolved through an identical policy but have never before been measured. We model this restriction on quantum measurements by allowing any replica observed in the anneal to be consumed as training data for the RL algorithm only once. We simulate this behaviour by keeping track of the policy decisions (the changes in inverse temperature) in an action buffer as we play through each episode. When a set of \(N_{\text{reps}}\) replicas are measured, they are consumed and the system is reset to a new set of initial conditions, as if it was a new episode. The actions held in the buffer are replayed on the new replicas.
+
+In this situation, the agent cannot base its decision on any replica-specific temporal correlations between given measurements; this should not be a problem early in each episode, as the correlation time scale of a hot system is very short, and the system, even under nondestructive observation, would have evolved sufficiently, in the time window between steps, to be uncorrelated. However, as the system cools, the correlation time scale increases exponentially, and destructive observation prevents the agent from relying on temporal correlations of any given replica.
+
+We evaluate an agent trained in this "quantum-inspired" way and plot its performance alongside the nondestructive (i.e., classical) case in Figure 4d. In the case of destructive observation, the agent performs marginally less well than the nondestructive case, but still performs better than SA in most cases. As it is a more complicated task to make observations when the system is temporally uncorrelated, it is understandable that the performance would be inferior to the nondestructive case. Nonetheless, RL is capable of outperforming SA in both the destructive and nondestructive cases.
+
+The relative performance in terms of computational demand between destructive observation and SA alludes to an important future direction in the field of RL, especially when applied to physical systems where observation is destructive, costly, and altogether difficult. With destructive observations, \(N_{\text{steps}}\) systems must be initialized and then evolved together under the same policy. Each copy is consumed one by one, as observations are required for decision making, thus incurring an unavoidable \(N_{\text{steps}}^{2}/2\) penalty in the destructive case. In this sense, it is difficult to consider RL to be superior; prescheduled SA simply does not require observation. However, if the choice to observe were to be incorporated into the action set of the RL algorithm, the agent would choose when observation would be necessary.
+
+For example, in the systems presented in this work, the correlation time of the observations is initially small; the temperatures are high, and frequent observations are required to guide the system through phase space. As the system cools, however, the correlation time grows exponentially, and the observations become much more similar to each previous observation; in this case, it would be beneficial to forgo some expensive observations, as the system would not be evolving substantially. With such a scheme, RL stands a better chance at achieving greater performance.
+
+### Policy analysis
+
+To glean some understanding into what the RL agent is learning, we train an additional model on a well-understood Hamiltonian, the ferromagnetic Ising model of size \(16\times 16\). In this case, the temperatures are initialized randomly (as in the WSC model). This model is the extreme case of a spin glass, with all \(J_{ij}=1\). In Figure 6a, we display the density of states \(g(M,E)\) of the Ising model, plotted in phase space, with axes of magnetization per spin (\(M/L\)) and energy per spin (\(E/L\)). The density of states is greatest in the high-entropy \(M=E=0\) region, and lowest in the low-entropy "corners". We show the spin configurations at the three corners ("checkerboard", spin-up, and spin-down) for clarity. The density of states is obtained numerically using Wang-Landau sampling [59]. Magnetization and energy combinations outside of the dashed "triangle" are impossible.
+
+In Figure 6b, we plot a histogram of the average value function \(V(s_{t})\) on the phase plane, as well as three trajectories. Note that since each observation \(s_{t}\) is composed of \(N_{\text{reps}}\) replicas, we count each observation as \(N_{\text{reps}}\) separate points on the phase plot when computing the histogram, each with an identical contribution of \(V(s_{t})\) to the average. As expected, the learned value function trends higher toward the two global energy minima. The lowest values are present in the initialization region (the high-energy band along the top). We expand two regions of interest in Figure 6c-d. In Figure 6d, we can see that the global minimum is assigned the highest value; this is justifiable in that if the agent reaches this point, it is likely to remain here and reap a high reward so long as the agent keeps the temperature low for the remainder of the episode.
+
+In Figure 6c, we identify four noteworthy energy-magnetization combinations, using asterisks. These four energy-magnetization combinations have identical energies, with increasing magnetization, and correspond to banded spin structures of decreasing width (four example spin configurations are shown). The agent learns to assign a higher value to the higher-magnetization structures, even though the energy, which is the true measure of "success", is identical. This is because the higher-magnetization bands are closer to the right-most global minimum in action space, that is, the agent can traverse from the small-band configuration to the ground state in fewer spin flips than if traversing from the wide-band configurations.
+
+In Figure 6e, we plot a histogram of the average action taken at each point in phase space. The upper high-energy band exhibits more randomness in the actions chosen, as this is the region in which the system lands upon initialization. When initialized, the temperature is at a randomly drawn value, and sometimes the agent must first heat the system to escape a local minimum before then cooling, and thus the first action is, on average, of very low magnitude. As the agent progresses toward the minimum, the agent becomes more aggressive in cooling the system, thereby thermally trapping itself in lower energy states.
+
+### Scaling and time to solution
+
+Figure 4d indicates that both nondestructive and destructive RL perform substantially better not only in absolute terms, but also in terms of scaling. It is important to note that we have specifically chosen a neural network architecture (convolutional) that scales linearly with system size, and have trained each model for the same number of episodes, each consisting of the same number of sweeps. The computation time for each sweep scales linearly with the system size, and thus the training time of our RL models scales linearly with system size. Using RL does indeed impose an additional inference cost, as the observation must be processed by the neural network; on the \(L=10^{2}\) system, inference takes one-third the amount of time as does each episode step. However, this cost has not been optimized, and could significantly be lowered through optimization of the neural network inference or even by offloading the policy network onto specialized hardware designed for inference.
+
+## VI Conclusion
+
+In this work, we show that reinforcement learning is a viable method for learning dynamic control schemes for the task of simulated annealing (SA). We show that, on a simple spin model, a reinforcement learning (RL) agent is capable of devising a temperature control scheme that can consistently escape a local minimum, and then anneal to the ground state. It arrives at a policy that generalizes to a range of initialization temperatures; in all cases, it learns to cool the system. However, if the initial temperature is too low, the RL agent learns to first increase the temperature to provide sufficient thermal energy to escape the local minimum. It achieves this without being provided explicit knowledge of the temperature.
+
+We then demonstrate that the RL agent is capable of learning a policy that can generalize to an entire class of Hamiltonians, and that the problem need not be re
+
+Figure 6: We train an agent on a special case of the spin-glass Hamiltonians: the \(16\times 16\) ferromagnetic Ising model where all couplings \(J_{ij}=1\). (a) We plot the density of states \(\log(g(M,E))\) for the \(16\times 16\) Ising model in the phase space of energy and magnetization, sampled numerically using the Wang–Landau algorithm [59], and indicate four of the novel high- and low-energy spin configurations on a grid. (b) For the trained model, we plot the average of the learned value function \(V(s_{t})\) for each possible energy–magnetization pair. Additionally, we plot the trajectories of the first replica for three episodes of annealing to demonstrate the path through phase space the algorithm learns to take. In (c) and (d), we enlarge two high-value regions of interest. In (e), we plot the average action taken at each point in phase space, as well as the same two trajectories plotted in (b).
+
+stricted to a single set of couplings. By training multiple RL agents on increasing numbers of variables (increasing lattice sizes), we investigate the scaling of the RL algorithm and find that it outperforms a classic SA schedule both in absolute terms and in terms of its scaling.
+
+Our technique is not limited to the system sizes we present in this work; larger system sizes are also within its reach. At some point, as the size of the system increases, correlation times in the underlying Metropolis-Hastings simulation become larger than the intervals between observations, and the number of sweeps must be increased. Additionally, we have specifically chosen a neural network architecture that scales linearly with system size (convolutional neural networks) as opposed to traditional multi-layer perceptron networks that scale exponentially. In fact, the entire procedure scales at most polynomially with system size.
+
+We analyze the value function that the agent learns and see that it attributes an intuitive representation of value to specific regions of phase space.
+
+We discuss the nature of RL in the physical sciences, specifically in situations where observing systems is destructive ("destructive observation") or costly (e.g., performing quantum computations where observations collapse the wavefunction, or conducting chemical analysis techniques that destroy a sample material). We demonstrate that our implementation of RL is capable of performing well in a destructive observation situation, albeit inefficiently. We propose that the future of physical RL (i.e., RL in the physical sciences) will be one of "controlled observation", where the algorithm can choose when an observation is necessary, minimizing the inherent costs incurred when observations are expensive, slow, or difficult.
+
+## VII Correspondence
+
+Requests for materials can be made to any of the authors. The code and data is freely available at the source provided below.
+
+## VIII Acknowledgements
+
+I. T. acknowledges support from NSERC. K. M. acknowledges support from Mitacs. The authors would like to thank Bruce Kravenhoff for valuable discussions in the early stages of the project, and would like to thank Marko Bucyk for reviewing and editing the manuscript.
+
+## IX Statement of contributions
+
+All authors contributed to the ideation and design of the research. K. M. developed and ran the computational experiments, and wrote the initial draft of the the manuscript. P. R. and I. T. jointly supervised this work and revised the manuscript.
+
+## X Data availability
+
+The test data sets necessary to reproduce these findings are available at [https://doi.org/10.5281/zenodo.3897413](https://doi.org/10.5281/zenodo.3897413).
+
+## XI Code availability
+
+The code necessary to reproduce these findings is available at [https://doi.org/10.5281/zenodo.3897413](https://doi.org/10.5281/zenodo.3897413).
+
+## References
+
+* Kirkpatrick _et al._ [1983]Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. Optimization by Simulated Annealing. _Science_**220**, 671-680 (1983). URL [http://www.sciencemag.org/cgi/doi/10.1126/science.220.4598.671](http://www.sciencemag.org/cgi/doi/10.1126/science.220.4598.671).
+* Barahona [1982]Barahona, F. On the computational complexity of Ising spin glass models. _Journal of Physics A: Mathematical and General_**15**, 3241-3253 (1982). URL [http://stacks.iop.org/0305-4470/15/i=10/a=028?key=crossref.ife57df46674a7c759374b69321415b44](http://stacks.iop.org/0305-4470/15/i=10/a=028?key=crossref.ife57df46674a7c759374b69321415b44).
+* Sherrington & Kirkpatrick [1975]Sherrington, D. & Kirkpatrick, S. Solvable Model of a Spin-Glass. _Physical Review Letters_**35**, 1792-1796 (1975). URL [https://link.aps.org/doi/10.1103/PhysRevLett.35.1792](https://link.aps.org/doi/10.1103/PhysRevLett.35.1792).
+* Ising [1925]Ising, E. Beitrag zur Theorie des Ferromagnetismus. _Zeitschrift fur Physik_**31**, 253-258 (1925).
+* Onsager [1944]Onsager, L. Crystal statistics. I. A two-dimensional model with an order-disorder transition. _Physical Review_**65**, 117-149 (1944).
+* Ferdinand & Fisher [1969]Ferdinand, A. E. & Fisher, M. E. Bounded and Inhomogeneous Ising Models. I. Specific-Heat Anomaly of a Finite Lattice. _Physical Review_**185**, 832-846 (1969). URL [https://link.aps.org/doi/10.1103/PhysRev.185.832](https://link.aps.org/doi/10.1103/PhysRev.185.832).
+* Lucas [2014]Lucas, A. Ising formulations of many NP problems. _Frontiers in Physics_**2**, 1-14 (2014).
+* Hastings _et al._ [1970]Hastings, B. Y. W. K. Monte Carlo sampling methods using Markov chains and their applications 97-109 (1970).
+* Metropolis _et al._ [1953]Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. & Teller, E. Equation of State Calculations by Fast Computing Machines. _The Journal of Chemical Physics_**21**, 1087-1092 (1953). URL [http://aip.scitation.org/doi/10.1063/1.1699114](http://aip.scitation.org/doi/10.1063/1.1699114).
+* Kirkpatrick [1984]Kirkpatrick, S. Optimization by simulated annealing: Quantitative studies. _Journal of Statistical Physics_**34**, 975-986 (1984). URL [http://link.springer.com/10.1007/BF01009452](http://link.springer.com/10.1007/BF01009452).
+
+* (11) van Laarhoven, P. J. M. & Aarts, E. H. L. _Simulated Annealing: Theory and Applications_ (Springer Netherlands, Dordrecht, 1987). URL [http://link.springer.com/10.1007/978-94-015-7744-1](http://link.springer.com/10.1007/978-94-015-7744-1).
+* (12) Stander, J. & Silverman, B. W. Temperature schedules for simulated annealing. _Statistics and Computing_**4**, 21-32 (1994).
+* (13) Heim, B., Ronnow, T. F., Isakov, S. V. & Troyer, M. Quantum versus classical annealing of ising spin glasses. _Science_**348**, 215-217 (2015). URL [https://science.sciencemag.org/content/348/6231/215](https://science.sciencemag.org/content/348/6231/215).
+* (14) Bounds, D. G. New optimization methods from physics and biology. _Nature_**329**, 215-219 (1987).
+* (15) Farhi, E. _et al._ A Quantum Adiabatic Evolution Algorithm Applied to Random Instances of an NP-Complete Problem. _Science_**292**, 472-475 (2001). eprint 0104129.
+* Atomic, Molecular, and Optical Physics_**86**, 1-8 (2012). eprint 1207.1712.
+* (17) Boixo, S. _et al._ Evidence for quantum annealing with more than one hundred qubits. _Nature Physics_**10**, 218-224 (2014). eprint 1304.4595.
+* (18) Bian, Z. _et al._ Discrete optimization using quantum annealing on sparse Ising models. _Frontiers in Physics_**2**, 1-10 (2014).
+* (19) Venturelli, D., Marchand, D. J. J. & Rojo, G. Quantum Annealing Implementation of Job-Shop Scheduling 1-15 (2015). URL [http://arxiv.org/abs/1506.08479.1506.08479](http://arxiv.org/abs/1506.08479.1506.08479).
+* (20) Ray, P., Chakrabarti, B. K. & Chakrabarti, A. Sherrington-Kirkpatrick model in a transverse field: Absence of replica symmetry breaking due to quantum fluctuations. _Physical Review B_**39**, 11828-11832 (1989).
+* Condensed Matter and Materials Physics_**66**, 1-8 (2002).
+* (22) Santoro, G. E. Theory of Quantum Annealing of an Ising Spin Glass. _Science_**295**, 2427-2430 (2002). URL [http://www.sciencemag.org/cgi/doi/10.1126/science.1068774.0205280](http://www.sciencemag.org/cgi/doi/10.1126/science.1068774.0205280).
+* (23) Finnila, A., Gomez, M., Sebenik, C., Stenson, C. & Doll, J. Quantum annealing: A new method for minimizing multidimensional functions. _Chemical Physics Letters_**219**, 343-348 (1994). URL [https://linkinghub.elsevier.com/retrieve/pii/0009261494001170](https://linkinghub.elsevier.com/retrieve/pii/0009261494001170).
+* Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics_**58**, 5355-5363 (1998). eprint 9804280.
+* Condensed Matter and Materials Physics_**81**, 1-19 (2010). eprint 0909.4321.
+* Condensed Matter and Materials Physics_**82**, 1-15 (2010). eprint 1004.1628.
+* (27) Johnson, M. W. _et al._ Quantum annealing with manufactured spins. _Nature_**473**, 194-198 (2011).
+* (28) McGeoch, C. C. & Wang, C. Experimental evaluation of an adiabiatic quantum system for combinatorial optimization. _Proceedings of the ACM International Conference on Computing Frontiers, CF 2013_ (2013).
+* (29) Ikeda, K., Nakamura, Y. & Humble, T. S. Application of Quantum Annealing to Nurse Scheduling Problem. _Scientific Reports_**9**, 1-10 (2019). eprint 1904.12139.
+* (30) Dickson, N. G. _et al._ Thermally assisted quantum annealing of a 16-qubit problem. _Nature Communications_**4**, 1-6 (2013).
+* (31) Okada, S., Ohzeki, M. & Tanaka, K. The efficient quantum and simulated annealing of Potts models using a half-hot constraint (2019). URL [http://arxiv.org/abs/1904.01522](http://arxiv.org/abs/1904.01522). eprint 1904.01522.
+* Statistical, Nonlinear, and Soft Matter Physics_**71**, 1-10 (2005). eprint 0502468.
+* (33) Tsukamoto, S., Takatsu, M., Matsubara, S. & Tamura, H. An accelerator architecture for combinatorial optimization problems. _Fujitsu Scientific and Technical Journal_**53**, 8-13 (2017).
+* (34) Inagaki, T. _et al._ A coherent Ising machine for 2000-node optimization problems. _Science_**354**, 603-606 (2016).
+* (35) Leleu, T., Yamamoto, Y., McMahon, P. L. & Aihara, K. Destabilization of local minima in analog spin systems by correction of amplitude heterogeneity. _Phys. Rev. Lett._**122**, 040607 (2019). URL [https://link.aps.org/doi/10.1103/PhysRevLett.122.040607](https://link.aps.org/doi/10.1103/PhysRevLett.122.040607).
+* (36) Tiunov, E. S., Ulanov, A. E. & Lvovsky, A. I. Annealing by simulating the coherent ising machine. _Opt. Express_**27**, 10288-10295 (2019). URL [http://www.opticsexpress.org/abstract.cfm?URI=oe-27-7-10288](http://www.opticsexpress.org/abstract.cfm?URI=oe-27-7-10288).
+* (37) Farhi, E., Goldstone, J. & Gutmann, S. A Quantum Approximate Optimization Algorithm 1-16 (2014). URL [http://arxiv.org/abs/1411.4028](http://arxiv.org/abs/1411.4028). eprint 1411.4028.
+* (38) Farhi, E., Goldstone, J., Gutmann, S. & Zhou, L. The Quantum Approximate Optimization Algorithm and the Sherrington-Kirkpatrick Model at Infinite Size 1-31 (2019). URL [http://arxiv.org/abs/1910.08187](http://arxiv.org/abs/1910.08187).
+* (39) Sutton, R. & Barto, A. _Reinforcement Learning: An Introduction_ (The MIT Press, 2018), second edn. URL [http://incompleteideas.net/book/the-book-2nd.htmlhttps://www.cambridge.org/core/product/identifier/S0263574799271172/type/journal](http://incompleteideas.net/book/the-book-2nd.htmlhttps://www.cambridge.org/core/product/identifier/S0263574799271172/type/journal){_}article[http://ieeexplore.ieee.org/document/712192/](http://ieeexplore.ieee.org/document/712192/).
+* (40) Berner, C. _et al._ Data 2 with Large Scale Deep Reinforcement Learning (2019).
+* (41) Zhang, Z. _et al._ Hierarchical Reinforcement Learning for Multi-agent MOBA Game (2019). URL [http://arxiv.org/abs/1901.08004](http://arxiv.org/abs/1901.08004). eprint 1901.08004.
+* (42) Vinyals, O. _et al._ Grandmaster level in StarCraft II using multi-agent reinforcement learning. _Nature_ (2019). URL [http://www.nature.com/articles/s41586-019-1724-z](http://www.nature.com/articles/s41586-019-1724-z).
+* (43) Mnih, V. _et al._ Playing Atari with Deep Reinforcement Learning. _arXiv preprint_ 1-9 (2013). URL [http://arxiv.org/abs/1312.5602](http://arxiv.org/abs/1312.5602). eprint 1312.5602.
+* (44) Silver, D. _et al._ Mastering the game of Go with deep neural networks and tree search. _Nature_**529**, 484-489 (2016). URL [http://dx.doi.org/10.1038/nature16961http://www.nature.com/doifinder/10.1038/nature16961](http://dx.doi.org/10.1038/nature16961http://www.nature.com/doifinder/10.1038/nature16961)* Silver _et al._ [2017] Silver, D. _et al._ Mastering the game of Go without human knowledge. _Nature_**550**, 354-359 (2017). URL [http://www.nature.com/doifinder/10.1038/nature24270](http://www.nature.com/doifinder/10.1038/nature24270).
+* Silver _et al._ [2018] Silver, D. _et al._ A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. _Science_**362**, 1140-1144 (2018).
+* Agostinelli _et al._ [2019] Agostinelli, F., McAleer, S., Shmakov, A. & Baldi, P. Solving the Rubik's cube with deep reinforcement learning and search. _Nature Machine Intelligence_**1**, 356-363 (2019). URL [http://www.nature.com/articles/s42256-019-0070-z](http://www.nature.com/articles/s42256-019-0070-z).
+* OpenAI _et al._ [2019] OpenAI _et al._ Solving Rubik's Cube with a Robot Hand 1-51 (2019). URL [http://arxiv.org/abs/1910.07113](http://arxiv.org/abs/1910.07113).
+* Schulman _et al._ [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal Policy Optimization Algorithms 1-12 (2017). URL [http://arxiv.org/abs/1707.06347](http://arxiv.org/abs/1707.06347).
+* Hill _et al._ [2018] Hill, A. _et al._ Stable baselines. [https://github.com/hill-a/stable-baselines](https://github.com/hill-a/stable-baselines) (2018).
+* Brockman _et al._ [2016] Brockman, G. _et al._ OpenAI Gym 1-4 (2016). URL [http://arxiv.org/abs/1606.01540](http://arxiv.org/abs/1606.01540). 1606.01540.
+* Schulman _et al._ [2015] Schulman, J., Levine, S., Abbeel, P., Jordan, M. & Moritz, P. Trust region policy optimization. In _International conference on machine learning_, 1889-1897 (2015).
+* Kakade & Langford [2002] Kakade, S. & Langford, J. Approximately optimal approximate reinforcement learning. In _ICML_, vol. 2, 267-274 (2002).
+* Hochreiter & Schmidhuber [1997] Hochreiter, S. & Schmidhuber, J. Long Short-Term Memory. _Neural Computation_**9**, 1735-1780 (1997). URL [http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735](http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735).
+* Aramon _et al._ [2019] Aramon, M. _et al._ Physics-inspired optimization for quadratic unconstrained problems using a digital Annealer. _Frontiers in Physics_**7** (2019). 1806.08815.
+* Bunyk _et al._ [2014] Bunyk, P. I. _et al._ Architectural Considerations in the Design of a Superconducting Quantum Annealing Processor. _IEEE Transactions on Applied Superconductivity_**24**, 1-10 (2014). 1401.5504.
+* Liers _et al._ [2005] Liers, F., Junger, M., Reinelt, G. & Rinaldi, G. Computing Exact Ground States of Hard Ising Spin Glass Problems by Branch-and-Cut. _New Optimization Algorithms in Physics_ 47-69 (2005).
+* Junger [2014] Junger, M. Spin Glass Server ([https://informatik.uni-koeln.de/spinglass/](https://informatik.uni-koeln.de/spinglass/)). URL [https://informatik.uni-koeln.de/spinglass/](https://informatik.uni-koeln.de/spinglass/).
+* Wang & Landau [2001] Wang, F. & Landau, D. P. Efficient, multiple-range random walk algorithm to calculate the density of states. _Physical Review Letters_**86**, 2050-2053 (2001). 0011174.