atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[The Hard Way Is Easier]]></title>
  <link href="http://cs50Mu.github.io/atom.xml" rel="self"/>
  <link href="http://cs50Mu.github.io/"/>
  <updated>2021-02-05T14:15:19+08:00</updated>
  <id>http://cs50Mu.github.io/</id>
  <author>
    <name><![CDATA[linuxfish]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[「Applying UML and Patterns」读书笔记]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-applying-uml-and-patterns/"/>
    <updated>2021-02-05T12:08:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-applying-uml-and-patterns</id>
    <content type="html"><![CDATA[<h2>「Applying UML and Patterns」读书笔记</h2>

<h3>Inception</h3>

<p>前期评估</p>

<p>the idea is to do just enough investigation to form a rational, justifiable opinion of the overall purpose and feasibility of the potential new system, and decide if it is worthwhile to invest in deeper exploration</p>

<p><strong>It is to decide if the project is worth a serious investigation (during elaboration), not to do that investigation.</strong></p>

<h4>Use Cases</h4>

<p>Informally, use cases are text stories of some actor using a system to meet goals.</p>

<p>the essence is discovering and recording functional requirements by writing stories of using a system to fulfill user goals; that is, cases of use</p>

<h3>Elaboration Iteration 1Basics &ndash; Domain Models</h3>

<h4>What is a Domain Model?</h4>

<p>A domain model is a visual representation of conceptual classes or real-situation objects in a domain.</p>

<p>Domain models have also been called conceptual models (the term used in the first edition of this book), domain object models, and analysis object models.</p>

<p>Applying UML notation, a domain model is illustrated <strong>with a set of class diagrams in which no operations (method signatures) are defined.</strong> It provides a conceptual perspective. It may show:</p>

<ul>
<li>domain objects or conceptual classes</li>
<li>associations between conceptual classes</li>
<li>attributes of conceptual classes</li>
</ul>


<p>A domain model shows real-situation conceptual classes, <strong>not software classes.</strong></p>

<h4>How to Find Conceptual Classes?</h4>

<ul>
<li>Reuse or modify existing models.</li>
<li>Use a category list. 一些默认建议，非常好</li>
<li>Finding Conceptual Classes with Noun Phrase Identification. 通过名词来找灵感

<ul>
<li><strong>use cases</strong> are one rich source to mine for noun phrase identification.</li>
</ul>
</li>
</ul>


<blockquote><p>Perhaps the most common mistake when creating a domain model is to represent something as an attribute when it should have been a conceptual class.</p></blockquote>

<p>If we do not think of some conceptual class X as a number or text in the real world, X is probably a conceptual class, not an attribute. 类比现实世界</p>

<h4>Why Use &lsquo;Description&rsquo; Classes?</h4>

<p><img src="http://cs50Mu.github.io/images/post/description-class.jpg" alt="" /></p>

<p>item的描述是一直存在的，但item可能会卖光</p>

<p>The need for description classes is common in sales, product, and service domains. It is also common in manufacturing, which requires a description of a manufactured thing that is distinct from the thing itself.</p>

<h4>Associations</h4>

<p>An association is a relationship between classes (more precisely, instances of those classes) that indicates some meaningful and interesting connection</p>

<p>Multiplicity defines how many instances of a class A can be associated with one instance of a class B</p>

<p><img src="http://cs50Mu.github.io/images/post/association-multiplicity.jpg" alt="" /></p>

<h4>Attributes</h4>

<p>An attribute is a logical data value of an object.</p>

<p>Informally, most attribute types should be what are often thought of as &ldquo;primitive&rdquo; data types, such as numbers and booleans. The type of an attribute should not normally be a complex domain concept, such as a Sale or Airport. 类属性应该是基础类型，不应该是复杂的自定义领域类型（这种应该用Association来表达）</p>

<p>Relate conceptual classes with an association, not with an attribute.</p>

<h4>Conclusion: Is the Domain Model Correct?</h4>

<p>There is no such thing as a single correct domain model. All models are approximations of the domain we are attempting to understand; the domain model is primarily a tool of understanding and communication among a particular group. A useful domain model captures the essential abstractions and information required to understand the domain in the context of the current requirements, and aids people in understanding the domainits concepts, terminology, and relationships. 没有“正确”的模型</p>

<h3>Elaboration Iteration 1Basics &ndash; System Sequence Diagrams</h3>

<h4>What are System Sequence Diagrams?</h4>

<p>A system sequence diagram is a picture that shows, for one particular scenario of a use case, the events that external actors generate, their order, and inter-system events. All systems are treated as a <strong>black box</strong>; the emphasis of the diagram is events that cross the system boundary from actors to systems.</p>

<p>System behavior is a description of what a system does, without explaining how it does it.  用于描述系统是做什么的，而不是如何做</p>

<h3>Elaboration Iteration 1Basics &ndash; Operation Contracts</h3>

<h3>Elaboration Iteration 1Basics &ndash; Requirements to DesignIteratively</h3>

<p>Iteratively Do the Right Thing, Do the Thing Right 做对的事 和 把事做对</p>

<p>The requirements and object-oriented analysis has focused on learning to do the right thing; that is, understanding some of the outstanding goals for the case studies, and related rules and constraints. By contrast, the following design work will stress do the thing right; that is, skillfully designing a solution to satisfy the requirements for this iteration.</p>

<h3>Elaboration Iteration 1Basics &ndash; Logical Architecture and UML Package Diagrams</h3>

<h4>What is the Logical Architecture? And Layers?</h4>

<p>The logical architecture is the large-scale organization of the software classes into packages (or namespaces), subsystems, and layers. <strong>It&rsquo;s called the logical architecture because there&rsquo;s no decision about how these elements are deployed across different operating system processes or across physical computers in a network</strong></p>

<p>A layer is a very coarse-grained grouping of classes, packages, or subsystems that has cohesive responsibility for a major aspect of the system. Also, layers are organized such that &ldquo;higher&rdquo; layers (such as the UI layer) call upon services of &ldquo;lower&rdquo; layers, but not normally vice versa.</p>

<p>Typically layers in an OO system include:</p>

<ul>
<li>User Interface.</li>
<li>Application Logic and Domain Objects 领域层</li>
<li>Technical Services

<ul>
<li>general purpose objects and subsystems that provide supporting technical services, such as interfacing with a database or error logging.</li>
<li>These services are usually application-independent and reusable across several systems.</li>
</ul>
</li>
</ul>


<blockquote><p>Guideline: Design with Layers</p></blockquote>

<p>Applied to information systems, <strong>typical layers</strong> are illustrated and explained as follows:</p>

<ul>
<li>UI</li>
<li>Application

<ul>
<li>handles presentation layer request</li>
<li>workflow</li>
<li>session state</li>
</ul>
</li>
<li>Domain (aka Business, Application logic)

<ul>
<li>implementation of domain rules</li>
</ul>
</li>
<li>Business Infrasture

<ul>
<li>very general low-level business services</li>
<li>used in many business domains</li>
<li>eg, CurrencyConverter</li>
</ul>
</li>
<li>Technial Services

<ul>
<li>(relatively) high-level technical services and frameworks</li>
<li>eg, persistence, security</li>
</ul>
</li>
<li>Foundation

<ul>
<li>low-level technical services, utilities and frameworks</li>
<li>eg, data structures, threads, math, file, DB and network I/O</li>
</ul>
</li>
</ul>


<blockquote><p>Guideline: Cohesive Responsibilities; Maintain a Separation of Concerns</p></blockquote>

<p>每层干自己该干的事</p>

<p>The responsibilities of the objects in a layer should be strongly related to each other and should not be mixed with responsibilities of other layers. For example, objects in the UI layer should focus on UI work, such as creating windows and widgets, capturing mouse and keyboard events, and so forth. Objects in the application logic or &ldquo;domain&rdquo; layer should focus on application logic, such as calculating a sales total or taxes, or moving a piece on a game board.</p>

<p>UI objects should not do application logic. For example, a Java Swing JFrame (window) object should not contain logic to calculate taxes or move a game piece. And on the other hand, application logic classes should not trap UI mouse or keyboard events. That would violate a clear separation of concerns and maintaining high cohesion basic architectural principles.</p>

<blockquote><p>How do we design the application logic with objects?</p></blockquote>

<p>To create software objects with names and information similar to the real-world domain, and <strong>assign application logic responsibilities to them</strong>. For example, in the real world of POS, there are sales and payments. So, in software, we create a Sale and Payment class, and give them application logic responsibilities.</p>

<blockquote><p>Guideline: The Model-View Separation Principle</p></blockquote>

<p>In this context, model is a synonym for the domain layer of objects (it&rsquo;s an old OO term from the late 1970s). View is a synonym for UI objects, such as windows, Web pages, applets, and reports.</p>

<p><strong>The Model-View Separation principle[2] states that model (domain) objects should not have direct knowledge of view (UI) objects, at least as view objects</strong>. So, for example, a Register or Sale object should not directly send a message to a GUI window object ProcessSaleFrame, asking it to display something, change color, close, and so forth.</p>

<p>Model-View-Controller (MVC): The Model is the Domain Layer, the View is the UI Layer, and the Controllers are the workflow objects in the Application layer. 至今看到的对MVC的最完美的诠释吧</p>

<h3>Elaboration Iteration 1Basics &ndash; On to Object Design</h3>

<h4>Designing Objects: What are Static and Dynamic Modeling?</h4>

<p>There are two kinds of object models: dynamic and static. <strong>Dynamic models, such as UML interaction diagrams (sequence diagrams or communication diagrams), help design the logic, the behavior of the code or the method bodies.</strong> They tend to be the more interesting, difficult, important diagrams to create. <strong>Static models, such as UML class diagrams, help design the definition of packages, class names, attributes, and method signatures (but not method bodies).</strong></p>

<p>People new to UML tend to think that the important diagram is the static-view class diagram, but in fact, most of the challenging, interesting, useful design work happens while drawing the UML dynamic-view interaction diagrams. It&rsquo;s during dynamic object modeling (such as drawing sequence diagrams) that &ldquo;the rubber hits the road&rdquo; in terms of really thinking through the exact details of what objects need to exist and how they collaborate via messages and methods. 重点：先做Dynamic Modeling（新手会误认为要先做Static Modeling）</p>

<p>What&rsquo;s important is knowing how to think and design in objects, and apply object design best-practice patterns, which is a very different and much more valuable skill than knowing UML notation. 重点是OO设计，而不是UML</p>

<h3>Elaboration Iteration 1Basics &ndash; UML Interaction Diagrams</h3>

<p>The term interaction diagram is a generalization of two more specialized UML diagram types:</p>

<ul>
<li>sequence diagrams

<ul>
<li>illustrate interactions in a kind of fence format, in which each new object is added to the right</li>
</ul>
</li>
<li>communication diagrams

<ul>
<li>illustrate object interactions in a graph or network format, in which objects can be placed anywhere on the diagram</li>
</ul>
</li>
</ul>


<p><img src="http://cs50Mu.github.io/images/post/sequence-diagram.jpg" alt="" />
<img src="http://cs50Mu.github.io/images/post/communication-diagram.jpg" alt="" /></p>

<h3>Elaboration Iteration 1Basics &ndash; UML Class Diagrams</h3>

<h3>Elaboration Iteration 1Basics &ndash; GRASP: Designing Objects with Responsibilities</h3>

<p><strong>The critical design tool for software development is a mind well educated in design principles.</strong> It is not the UML or any other technology.</p>

<blockquote><p>What Are Inputs to Object Design?</p></blockquote>

<p>开始设计的前提是已做好需求分析</p>

<blockquote><p>Responsibilities and Responsibility-Driven Design</p></blockquote>

<p>Basically, these responsibilities are of the following two types: doing and knowing.</p>

<p>Doing responsibilities of an object include:</p>

<ul>
<li>doing something itself, such as creating an object or doing a calculation</li>
<li>initiating action in other objects</li>
<li>controlling and coordinating activities in other objects</li>
</ul>


<p>Knowing responsibilities of an object include:</p>

<ul>
<li>knowing about private encapsulated data</li>
<li>knowing about related objects</li>
<li>knowing about things it can derive or calculate</li>
</ul>


<blockquote><p>What are Patterns?</p></blockquote>

<p>In OO design, a pattern is a named description of a problem and solution that can be applied to new contexts; ideally, a pattern advises us on how to apply its solution in varying circumstances and considers the forces and trade-offs.</p>

<p>New pattern should be considered an oxymoron if it describes a new idea. The very term &ldquo;pattern&rdquo; suggests a long-repeating thing. <strong>The point of design patterns is not to express new design ideas. Quite the oppositegreat patterns attempt to codify existing tried-and-true knowledge, idioms, and principles</strong>; the more honed, old, and widely used, the better.</p>

<blockquote><p>A Short Example of Object Design with GRASP</p></blockquote>

<p>There are nine GRASP patterns:</p>

<ul>
<li>Creator</li>
</ul>


<p>关键词：mental model、low representational gap (LRG)</p>

<p>原则是要尽量降低人的心理模型跟design model之间的认知差异</p>

<ul>
<li>Information Expert</li>
</ul>


<p>Problem： What is a basic principle by which to assign responsibilities to objects?</p>

<p>Solution： Assign a responsibility to the class that has the information needed to fulfill it.</p>

<ul>
<li>Low Coupling</li>
</ul>


<p>Briefly and informally, coupling is a measure of how strongly one element is connected to, has knowledge of, or depends on other elements.</p>

<p>If there is coupling or dependency, then when the depended-upon element changes, the dependant may be affected. For example, a subclass is strongly coupled to a superclass. An object A that calls on the operations of object B has coupling to B&rsquo;s services.</p>

<p>It is not high coupling per se that is the problem; it is high coupling to elements that are unstable in some dimension, such as their interface, implementation, or mere presence.  不是耦合本身不好，而要看耦合的东西，若耦合的东西不稳定就不太好</p>

<ul>
<li>Controller</li>
</ul>


<p>Problem：What first object beyond the UI layer receives and coordinates (&ldquo;controls&rdquo;) a system operation?</p>

<p>Solution： Assign the responsibility to an object representing one of these choices:</p>

<ol>
<li>Represents the overall &ldquo;system,&rdquo; a &ldquo;root object,&rdquo; a device that the software is running within, or a major subsystem (these are all variations of a facade controller).</li>
<li>Represents a use case scenario within which the system operation occurs (a use case or session controller)</li>
</ol>


<p>有两种：</p>

<p>Facade controllers：门面，一切全包，比如TelecommSwitch、Phone、ChessGame</p>

<p>use case controller：按使用场景来分的，比如ProcessSaleHandler</p>

<ul>
<li>High Cohesion</li>
</ul>


<p>In software design a basic quality known as cohesion informally measures how <strong>functionally related</strong> the operations of a software element are, and also measures how much work a software element is doing. 做的事要有相关性，一个类的方法集中如果有很多关系不大的事情，则这个类不是cohesive的</p>

<p>As a simple contrasting example, an object Big with 100 methods and 2,000 source lines of code (SLOC) is doing a lot more than an object Small with 10 methods and 200 source lines. <strong>And if the 100 methods of Big are covering many different areas of responsibility (such as database access and random number generation), then Big has less focus or functional cohesion than Small.</strong> In summary, both the amount of code and the relatedness of the code are an indicator of an object&rsquo;s cohesion.</p>

<p>Problem: How to keep objects focused, understandable, and manageable, and as a side effect, support Low Coupling?</p>

<p>Solution: Assign responsibilities so that cohesion remains high. Use this to evaluate alternatives.</p>

<h3>Elaboration Iteration 1Basics &ndash; Object Design Examples with GRASP</h3>

<p>I wish to exhaustively illustrate that no &ldquo;magic&rdquo; is needed in object design</p>

<p>OO software design really can be <strong>more science than art</strong>, though there is plenty of room for creativity and elegant design.</p>

<p>实例讲解，非常详尽！</p>

<blockquote><p>The Command-Query Separation Principle</p></blockquote>

<p>命令（有side effect的操作，比如update create等）与 查询 分离的原则</p>

<p>要这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="c1">// style #1; used in the official solution</span>
</span><span class='line'><span class="kd">public</span> <span class="kt">void</span> <span class="nf">roll</span><span class="o">()</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'><span class="n">faceValue</span> <span class="o">=</span> <span class="c1">// random num generation</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'><span class="kd">public</span> <span class="kt">int</span> <span class="nf">getFaceValue</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>   <span class="k">return</span> <span class="n">faceValue</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>而不是这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="c1">// style #2; why is this poor?</span>
</span><span class='line'><span class="kd">public</span> <span class="kt">int</span> <span class="nf">roll</span><span class="o">()</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'><span class="n">faceValue</span> <span class="o">=</span> <span class="c1">// random num generation</span>
</span><span class='line'>   <span class="k">return</span> <span class="n">faceValue</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>为什么呢？</p>

<p>CQS is widely considered desirable in computer science theory <strong>because with it, you can more easily reason about a program&rsquo;s state without simultaneously modifying that state. And it makes designs simpler to understand and anticipate.</strong> For example, if an application consistently follows CQS, you know that a query or getter method isn&rsquo;t going to modify anything and a command isn&rsquo;t going to return anything. Simple pattern. This often turns out to be nice to rely on, as the alternative can be a nasty surpriseviolating the Principle of <strong>Least Surprise</strong> in software development</p>

<h3>Elaboration Iteration 1Basics &ndash; Designing for Visibility</h3>

<p>Visibility is the ability of one object to see or have reference to another</p>

<p>There are four common ways that visibility can be achieved from object A to object B:</p>

<ul>
<li>Attribute visibility B is an attribute of A.</li>
<li>Parameter visibility B is a parameter of a method of A.</li>
<li>Local visibility B is a (non-parameter) local object in a method of A.</li>
<li>Global visibility B is in some way globally visible.</li>
</ul>


<h3>Elaboration Iteration 1Basics &ndash; Mapping Designs to Code</h3>

<h3>Elaboration Iteration 1Basics &ndash; Test-Driven Development and Refactoring</h3>

<p>Refactoring [Fowler99] is a structured, disciplined method to rewrite or restructure existing code without changing its external behavior, applying small transformation steps combined with re- executing tests each step.</p>

<h3>Elaboration Iteration 2 More Patterns &ndash; UML Tools and UML as Blueprint</h3>

<h3>Elaboration Iteration 2 More Patterns &ndash; Quick Analysis Update</h3>

<h3>Elaboration Iteration 2 More Patterns &ndash; Iteration 2More Patterns</h3>

<h3>Elaboration Iteration 2 More Patterns &ndash; More Objects with Responsibilities</h3>

<p>GRASP patterns:</p>

<ul>
<li>Polymorphism</li>
</ul>


<p>多态</p>

<p>Alternatives based on type Conditional variation is a fundamental theme in programs. If a program is designed using if-then-else or case statement conditional logic, then if a new variation arises, it requires modification of the case logicoften in many places. This approach makes it difficult to easily extend a program with new variations because changes tend to be required in several placeswherever the conditional logic exists. if-else 问题</p>

<p>原则：不要用if-else或者case来解决，而是用多态来解决</p>

<ul>
<li>Pure Fabrication</li>
</ul>


<p>Object-oriented designs are sometimes characterized by implementing as software classes representations of concepts in the real-world problem domain to lower the representational gap</p>

<p>Problem: What object should have the responsibility, when you do not want to violate High Cohesion and Low Coupling, or other goals, but solutions offered by Expert (for example) are not appropriate? there are many situations in which assigning responsibilities only to domain layer software classes leads to problems in terms of poor cohesion or coupling, or low reuse potential. 没办法跟Domain model（现实模型）对应上的object怎么设计？</p>

<p>Solution: Assign a highly cohesive set of responsibilities to <strong>an artificial or convenience class that does not represent a problem domain concept something made up</strong>, to support high cohesion, low coupling, and reuse.</p>

<ul>
<li>Indirection</li>
</ul>


<p>Most problems in computer science can be solved by another level of indirection</p>

<ul>
<li>Protected Variations</li>
</ul>


<p>Problem: How to design objects, subsystems, and systems so that the variations or instability in these elements does not have an undesirable impact on other elements?</p>

<p>Solution: Identify points of predicted variation or instability; <strong>assign responsibilities to create a stable interface around them</strong>.</p>

<p>虚拟机也是一个应用这个原则的例子</p>

<blockquote><p>The Liskov Substitution Principle (LSP)</p></blockquote>

<p>LSP [Liskov88] formalizes the principle of protection against variations in different implementations of an interface, or subclass extensions of a superclass.</p>

<p>Informally, software (methods, classes, &hellip;) that refers to a type T (some interface or abstract superclass) should work properly or as expected with any substituted implementation or subclass of Tcall it S.</p>

<blockquote><p>Law of Demeter</p></blockquote>

<p>Don&rsquo;t Talk to Strangers</p>

<p>违反<code>Law of Demeter</code>的代码示例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kt">void</span> <span class="nf">doX</span><span class="o">()</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'><span class="n">F</span> <span class="n">someF</span> <span class="o">=</span> <span class="n">foo</span><span class="o">.</span><span class="na">getA</span><span class="o">().</span><span class="na">getB</span><span class="o">().</span><span class="na">getC</span><span class="o">().</span><span class="na">getD</span><span class="o">().</span><span class="na">getE</span><span class="o">().</span><span class="na">getF</span><span class="o">();</span>
</span><span class='line'><span class="c1">// ... </span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p><strong>The design is coupled to a particular structure of how objects are connected.</strong> The farther along a path the program traverses, the more fragile it is. Why? Because the object structure (the connections) may change.</p>

<p>Novice developers tend toward brittle designs, intermediate developers tend toward overly fancy and flexible, generalized ones (in ways that never get used). Expert designers choose with insight;  关键是这个度不好把握啊</p>

<blockquote><p>Open-Closed Principle</p></blockquote>

<p>OCP and PV are essentially two expressions of the same principle, with different emphasis</p>

<h3>Elaboration Iteration 2 More Patterns &ndash; Applying GoF Design Patterns</h3>

<blockquote><p>Adapter (GoF)</p></blockquote>

<p>Problem:  How to resolve incompatible interfaces, or provide a stable interface to similar components with different interfaces?</p>

<p>Solution: Convert the original interface of a component into another interface, through an intermediate adapter object.</p>

<blockquote><p>Factory</p></blockquote>

<p>The adapter raises a new problem in the design:</p>

<p>who creates the adapters? And how to determine which class of adapter to create, such as TaxMaster-Adapter or GoodAsGoldTaxProAdapter?</p>

<p>答案就是Factory</p>

<p>Problem: Who should be responsible for creating objects when there are special considerations, such as complex creation logic, a desire to separate the creation responsibilities for better cohesion, and so forth?</p>

<p>Solution: Create a Pure Fabrication object called a Factory that handles the creation.</p>

<p><img src="http://cs50Mu.github.io/images/post/factory.jpg" alt="" /></p>

<p>重点关注上图中，它是如何决定应该使用哪个Adapter的</p>

<p>Note that in the ServicesFactory, the logic to decide which class to create is resolved <strong>by reading in the class name from an external source (for example, via a system property if Java is used) and then dynamically loading the class.</strong> This is an example of a partial data-driven design.</p>

<blockquote><p>Singleton (GoF)</p></blockquote>

<p>The ServicesFactory raises another new problem in the design: Who creates the factory itself, and how is it accessed? 汗死。。。  按下葫芦起了瓢的感觉</p>

<p>观察</p>

<p>First, observe that only one instance of the factory is needed within the process. Second, quick reflection suggests that the methods of this factory may need to be called from various places in the code, as different places need access to the adapters for calling on the external services.</p>

<p>Problem: Exactly one instance of a class is allowedit is a &ldquo;singleton.&rdquo; Objects need a global and single point of access.</p>

<p>Solution: Define a static method of the class that returns the singleton.</p>

<p><img src="http://cs50Mu.github.io/images/post/singleton.jpg" alt="" /></p>

<blockquote><p>Strategy (GoF)</p></blockquote>

<p>Problem: How to design for varying, but related, algorithms or policies? How to design for the ability to change these algorithms or policies?</p>

<p>Solution: Define each algorithm/policy/strategy in a separate class, with a common interface.</p>

<blockquote><p>Composite (GoF)</p></blockquote>

<p>Problem：How to treat a group or composition structure of objects the same way (polymorphically) as a non-composite (atomic) object?</p>

<p>Solution: Define classes for composite and atomic objects so that they implement the same interface.</p>

<p>没有太理解清楚这个pattern</p>

<blockquote><p>Facade (GoF)</p></blockquote>

<p>Problem: A common, unified interface to a disparate set of implementations or interfacessuch as within a subsystemis required. There may be undesirable coupling to many things in the subsystem, or the implementation of the subsystem may change. What to do?</p>

<p>Solution: Define a single point of contact to the subsystema facade object that wraps the subsystem. This facade object presents a single unified interface and is responsible for collaborating with the subsystem components.</p>

<blockquote><p>Observer (Publish-Subscribe)</p></blockquote>

<p>Problem: Different kinds of subscriber objects are interested in the state changes or events of a publisher object, and want to react in their own unique way when the publisher generates an event. Moreover, the publisher wants to maintain low coupling to the subscribers. What to do?</p>

<p>Solution: Define a &ldquo;subscriber&rdquo; or &ldquo;listener&rdquo; interface. Subscribers implement this interface. The publisher can dynamically register subscribers who are interested in an event and notify them when an event occurs.</p>

<blockquote><p>Observer/Publish-Subscribe/Delegation Event Model (GoF)</p></blockquote>

<p>都是一个意思</p>

<p>Problem：Different kinds of subscriber objects are interested in the state changes or events of a publisher object, and want to react in their own unique way when the publisher generates an event. Moreover, the publisher wants to maintain low coupling to the subscribers. What to do?</p>

<p>Solution: Define a &ldquo;subscriber&rdquo; or &ldquo;listener&rdquo; interface. Subscribers implement this interface. The publisher can dynamically register subscribers who are interested in an event and notify them when an event occurs.</p>

<p>Observer provides a way to loosely couple objects in terms of communication. Publishers know about subscribers only through an interface, and subscribers can register (or de-register) dynamically with the publisher.</p>

<h3>Elaboration Iteration 3 &ndash; Intermediate Topics</h3>

<h3>Elaboration Iteration 3 &ndash; UML Activity Diagrams and Modeling</h3>

<p>A UML activity diagram shows sequential and parallel activities in a process. They are useful for modeling business processes, workflows, data flows, and complex algorithms.</p>

<p><img src="http://cs50Mu.github.io/images/post/activity-diagram.jpg" alt="" /></p>

<h3>Elaboration Iteration 3 &ndash; UML State Machine Diagrams and Modeling</h3>

<p>A UML state machine diagram, illustrates the interesting events and states of an object, and the behavior of an object in reaction to an event.</p>

<p>A state machine diagram shows the lifecycle of an object: what events it experiences, its transitions, and the states it is in between these events.</p>

<p><img src="http://cs50Mu.github.io/images/post/state-machine.jpg" alt="" /></p>

<h3>Elaboration Iteration 3 &ndash; Relating Use Cases</h3>

<h3>Elaboration Iteration 3 &ndash; More SSDs and Contracts</h3>

<h3>Elaboration Iteration 3 &ndash; Domain Model Refinement</h3>

<h3>Elaboration Iteration 3 &ndash; Architectural Analysis</h3>

<p>Architectural analysis can be viewed as a specialization of requirements analysis, with a focus on requirements that strongly influence the &ldquo;architecture.&rdquo; For example, identifying the need for a highly-secure system.</p>

<p>The essence of architectural analysis is to <strong>identify factors that should influence the architecture, understand their variability and priority, and resolve them.</strong> The difficult part is knowing what questions to ask, weighing the trade-offs, and knowing the many ways to resolve an architecturally significant factor, ranging from benign neglect, to fancy designs, to third-party products</p>

<ul>
<li>variation point

<ul>
<li>Variations in the existing current system or requirements, such as the multiple tax calculator interfaces that must be supported.</li>
</ul>
</li>
<li>evolution point

<ul>
<li>Speculative points of variation that may arise in the future, but which are not present in the existing requirements.</li>
</ul>
</li>
</ul>


<p>非功能性分析，比如安全、性能等</p>

<h3>Elaboration Iteration 3 &ndash; Logical Architecture Refinement</h3>

<p><img src="http://cs50Mu.github.io/images/post/view-of-layers.jpg" alt="" /></p>

<p>When the lower Application or Domain layer needs to communicate upward with the UI layer, it is usually via the Observer pattern.</p>

<p>it is not coupling per se that is a problem, but unnecessary coupling to variation and evolution points that are unstable and expensive to fix.</p>

<blockquote><p>Is the Application Layer Optional?</p></blockquote>

<p>If present, the Application layer contains objects responsible for knowing the session state of clients, mediating between the UI and Domain layers, and controlling the flow of work.</p>

<h3>Elaboration Iteration 3 &ndash; More Object Design with GoF Patterns</h3>

<blockquote><p>Handling Failure</p></blockquote>

<p>a common exception handling pattern:</p>

<ul>
<li>Convert Exceptions

<ul>
<li>Within a subsystem, avoid emitting lower level exceptions coming from lower subsystems or services. Rather, convert the lower level exception into one that is meaningful at the level of the subsystem. The higher level exception usually wraps the lower-level exception, and adds information, to make the exception more contextually meaningful to the higher level.</li>
</ul>
</li>
</ul>


<p>For example, the persistence subsystem catches a particular SQLException, and (assuming it can&rsquo;t handle it[2] ) throws a new DBUnavailableException, which contains the SQLException. Note that the DBProductAdapter is like a facade onto a logical subsystem for product information. Thus, the higher level DBProductAdapter (as the representative for a logical subsystem) catches the lower level DBUnavailableException and (assuming it can&rsquo;t handle it) throws a new ProductInfoUnavailableException, which wraps the DBUnavailableException.</p>

<ul>
<li><p>Name The Problem Not The Thrower</p>

<ul>
<li>What to call an exception? Assign a name that describes why the exception is being thrown, not the thrower. The benefit is that it makes it easier for the programmer to understand the problem, and it the highlights the essential similarity of many classes of exceptions (in a way that naming the thrower does not).</li>
</ul>
</li>
<li><p>Centralized Error Logging</p>

<ul>
<li>Use a Singleton-accessed central error logging object and report all exceptions to it. If it is a distributed system, each local singleton will collaborate with a central error logger.</li>
</ul>
</li>
<li><p>Error Dialog</p></li>
</ul>


<blockquote><p>Proxy</p></blockquote>

<p>Problem: Direct access to a real subject object is not desired or possible. What to do?</p>

<p>Solution: Add a level of indirection with a surrogate proxy object that implements the same interface as the subject object, and is responsibility for controlling or enhancing access to it.</p>

<blockquote><p>Abstract Factory</p></blockquote>

<p>Problem: How to create families of related classes that implement a common interface?</p>

<p>Solution: Define a factory interface (the abstract factory). Define a concrete factory class for each family of things to create. Optionally, define a true abstract class that implements the factory interface and provides common services to the concrete factories that extend it.</p>

<h3>Elaboration Iteration 3 &ndash; Package Design</h3>

<h3>Elaboration Iteration 3 &ndash; Designing a Persistence Framework with Patterns</h3>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Grafana源码分析]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/02/05/sourcecode-of-grafana/"/>
    <updated>2021-02-05T12:04:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/02/05/sourcecode-of-grafana</id>
    <content type="html"><![CDATA[<h2>Grafana源码分析</h2>

<h3>服务自动注入 / inject</h3>

<p>代码里的service使用了一个包来实现自动注入：<a href="https://github.com/facebookarchive/inject"><code>facebookarchive / inject</code></a></p>

<p>要实现这个自动，需要做以下准备工作：</p>

<p>打struct tag</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">type</span> <span class="nx">HTTPServer</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">log</span>           <span class="nx">log</span><span class="p">.</span><span class="nx">Logger</span>
</span><span class='line'>      <span class="nx">macaron</span>       <span class="o">*</span><span class="nx">macaron</span><span class="p">.</span><span class="nx">Macaron</span>
</span><span class='line'>      <span class="nx">context</span>       <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span>
</span><span class='line'>      <span class="nx">streamManager</span> <span class="o">*</span><span class="nx">live</span><span class="p">.</span><span class="nx">StreamManager</span>
</span><span class='line'>      <span class="nx">httpSrv</span>       <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Server</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">RouteRegister</span>        <span class="nx">routing</span><span class="p">.</span><span class="nx">RouteRegister</span>            <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">Bus</span>                  <span class="nx">bus</span><span class="p">.</span><span class="nx">Bus</span>                          <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">RenderService</span>        <span class="nx">rendering</span><span class="p">.</span><span class="nx">Service</span>                <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">Cfg</span>                  <span class="o">*</span><span class="nx">setting</span><span class="p">.</span><span class="nx">Cfg</span>                     <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">HooksService</span>         <span class="o">*</span><span class="nx">hooks</span><span class="p">.</span><span class="nx">HooksService</span>              <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">CacheService</span>         <span class="o">*</span><span class="nx">localcache</span><span class="p">.</span><span class="nx">CacheService</span>         <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">DatasourceCache</span>      <span class="nx">datasources</span><span class="p">.</span><span class="nx">CacheService</span>         <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">AuthTokenService</span>     <span class="nx">models</span><span class="p">.</span><span class="nx">UserTokenService</span>          <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">QuotaService</span>         <span class="o">*</span><span class="nx">quota</span><span class="p">.</span><span class="nx">QuotaService</span>              <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">RemoteCacheService</span>   <span class="o">*</span><span class="nx">remotecache</span><span class="p">.</span><span class="nx">RemoteCache</span>         <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">ProvisioningService</span>  <span class="nx">provisioning</span><span class="p">.</span><span class="nx">ProvisioningService</span> <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">Login</span>                <span class="o">*</span><span class="nx">login</span><span class="p">.</span><span class="nx">LoginService</span>              <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">License</span>              <span class="nx">models</span><span class="p">.</span><span class="nx">Licensing</span>                 <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">BackendPluginManager</span> <span class="nx">backendplugin</span><span class="p">.</span><span class="nx">Manager</span>            <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">PluginManager</span>        <span class="o">*</span><span class="nx">plugins</span><span class="p">.</span><span class="nx">PluginManager</span>           <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>      <span class="nx">SearchService</span>        <span class="o">*</span><span class="nx">search</span><span class="p">.</span><span class="nx">SearchService</span>            <span class="s">`inject:&quot;&quot;`</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>启动server时会build Service Graph:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 获取注册的服务，具体实现比较简单，分析略过</span>
</span><span class='line'><span class="nx">services</span> <span class="o">:=</span> <span class="nx">registry</span><span class="p">.</span><span class="nx">GetServices</span><span class="p">()</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">buildServiceGraph</span><span class="p">(</span><span class="nx">services</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">return</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'><span class="c1">// 看看它做了啥呢，这里就是关键了</span>
</span><span class='line'>  <span class="c1">// buildServiceGraph builds a graph of services and their dependencies.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">Server</span><span class="p">)</span> <span class="nx">buildServiceGraph</span><span class="p">(</span><span class="nx">services</span> <span class="p">[]</span><span class="o">*</span><span class="nx">registry</span><span class="p">.</span><span class="nx">Descriptor</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// Specify service dependencies.</span>
</span><span class='line'>      <span class="nx">objs</span> <span class="o">:=</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{}{</span>
</span><span class='line'>          <span class="nx">bus</span><span class="p">.</span><span class="nx">GetBus</span><span class="p">(),</span>
</span><span class='line'>          <span class="nx">s</span><span class="p">.</span><span class="nx">cfg</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">routing</span><span class="p">.</span><span class="nx">NewRouteRegister</span><span class="p">(</span><span class="nx">middleware</span><span class="p">.</span><span class="nx">RequestMetrics</span><span class="p">,</span> <span class="nx">middleware</span><span class="p">.</span><span class="nx">RequestTracing</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">localcache</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="mi">5</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">,</span> <span class="mi">10</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">s</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">service</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">services</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">objs</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">objs</span><span class="p">,</span> <span class="nx">service</span><span class="p">.</span><span class="nx">Instance</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">serviceGraph</span> <span class="nx">inject</span><span class="p">.</span><span class="nx">Graph</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Provide services and their dependencies to the graph.</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">obj</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">objs</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 奥秘在这里了</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">serviceGraph</span><span class="p">.</span><span class="nx">Provide</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">inject</span><span class="p">.</span><span class="nx">Object</span><span class="p">{</span><span class="nx">Value</span><span class="p">:</span> <span class="nx">obj</span><span class="p">});</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">errutil</span><span class="p">.</span><span class="nx">Wrapf</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="s">&quot;Failed to provide object to the graph&quot;</span><span class="p">)</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Resolve services and their dependencies.</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">serviceGraph</span><span class="p">.</span><span class="nx">Populate</span><span class="p">();</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">errutil</span><span class="p">.</span><span class="nx">Wrapf</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="s">&quot;Failed to populate service dependency&quot;</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>user login是如何实现的？</h3>

<p>有个middleware，在处理业务请求之前会先验证user</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// pkg/middleware/middleware.go</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">GetContextHandler</span><span class="p">(</span>
</span><span class='line'>  <span class="nx">ats</span> <span class="nx">models</span><span class="p">.</span><span class="nx">UserTokenService</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">remoteCache</span> <span class="o">*</span><span class="nx">remotecache</span><span class="p">.</span><span class="nx">RemoteCache</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">renderService</span> <span class="nx">rendering</span><span class="p">.</span><span class="nx">Service</span><span class="p">,</span>
</span><span class='line'><span class="p">)</span> <span class="nx">macaron</span><span class="p">.</span><span class="nx">Handler</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">macaron</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">ctx</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">models</span><span class="p">.</span><span class="nx">ReqContext</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">Context</span><span class="p">:</span>        <span class="nx">c</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">SignedInUser</span><span class="p">:</span>   <span class="o">&amp;</span><span class="nx">models</span><span class="p">.</span><span class="nx">SignedInUser</span><span class="p">{},</span>
</span><span class='line'>          <span class="nx">IsSignedIn</span><span class="p">:</span>     <span class="kc">false</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">AllowAnonymous</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">SkipCache</span><span class="p">:</span>      <span class="kc">false</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">Logger</span><span class="p">:</span>         <span class="nx">log</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">&quot;context&quot;</span><span class="p">),</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">orgId</span> <span class="o">:=</span> <span class="nb">int64</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">orgIdHeader</span> <span class="o">:=</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Req</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">&quot;X-Grafana-Org-Id&quot;</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">orgIdHeader</span> <span class="o">!=</span> <span class="s">&quot;&quot;</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">orgId</span><span class="p">,</span> <span class="nx">_</span> <span class="p">=</span> <span class="nx">strconv</span><span class="p">.</span><span class="nx">ParseInt</span><span class="p">(</span><span class="nx">orgIdHeader</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">64</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// the order in which these are tested are important</span>
</span><span class='line'>      <span class="c1">// look for api key in Authorization header first</span>
</span><span class='line'>      <span class="c1">// then init session and look for userId in session</span>
</span><span class='line'>      <span class="c1">// then look for api key in session (special case for render calls via api)</span>
</span><span class='line'>      <span class="c1">// then test if anonymous access is enabled</span>
</span><span class='line'>      <span class="c1">// 这个写法可以学习一下</span>
</span><span class='line'>      <span class="k">switch</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithRenderAuth</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">renderService</span><span class="p">):</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithApiKey</span><span class="p">(</span><span class="nx">ctx</span><span class="p">):</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithBasicAuth</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">orgId</span><span class="p">):</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithAuthProxy</span><span class="p">(</span><span class="nx">remoteCache</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">,</span> <span class="nx">orgId</span><span class="p">):</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithToken</span><span class="p">(</span><span class="nx">ats</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">,</span> <span class="nx">orgId</span><span class="p">):</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">initContextWithAnonymousUser</span><span class="p">(</span><span class="nx">ctx</span><span class="p">):</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">ctx</span><span class="p">.</span><span class="nx">Logger</span> <span class="p">=</span> <span class="nx">log</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">&quot;context&quot;</span><span class="p">,</span> <span class="s">&quot;userId&quot;</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">UserId</span><span class="p">,</span> <span class="s">&quot;orgId&quot;</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">OrgId</span><span class="p">,</span> <span class="s">&quot;uname&quot;</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Login</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">ctx</span><span class="p">.</span><span class="nx">Data</span><span class="p">[</span><span class="s">&quot;ctx&quot;</span><span class="p">]</span> <span class="p">=</span> <span class="nx">ctx</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Map</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// update last seen every 5min</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">ShouldUpdateLastSeenAt</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">ctx</span><span class="p">.</span><span class="nx">Logger</span><span class="p">.</span><span class="nx">Debug</span><span class="p">(</span><span class="s">&quot;Updating last user_seen_at&quot;</span><span class="p">,</span> <span class="s">&quot;user_id&quot;</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">UserId</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">bus</span><span class="p">.</span><span class="nx">Dispatch</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">models</span><span class="p">.</span><span class="nx">UpdateUserLastSeenAtCommand</span><span class="p">{</span><span class="nx">UserId</span><span class="p">:</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">UserId</span><span class="p">});</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">ctx</span><span class="p">.</span><span class="nx">Logger</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="s">&quot;Failed to update last_seen_at&quot;</span><span class="p">,</span> <span class="s">&quot;error&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>




<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// login接口的实现，它会调用下面的`AuthenticateUser`方法</span>
</span><span class='line'><span class="c1">// pkg/api/login.go</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">hs</span> <span class="o">*</span><span class="nx">HTTPServer</span><span class="p">)</span> <span class="nx">LoginPost</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">models</span><span class="p">.</span><span class="nx">ReqContext</span><span class="p">,</span> <span class="nx">cmd</span> <span class="nx">dtos</span><span class="p">.</span><span class="nx">LoginCommand</span><span class="p">)</span> <span class="nx">Response</span> <span class="p">{</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 这里的逻辑包括：验证user、写cookie（middleware.WriteSessionCookie）</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'><span class="c1">// pkg/login/auth.go</span>
</span><span class='line'><span class="c1">// AuthenticateUser authenticates the user via username &amp; password</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">AuthenticateUser</span><span class="p">(</span><span class="nx">query</span> <span class="o">*</span><span class="nx">models</span><span class="p">.</span><span class="nx">LoginUserQuery</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">validateLoginAttempts</span><span class="p">(</span><span class="nx">query</span><span class="p">.</span><span class="nx">Username</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">validatePasswordSet</span><span class="p">(</span><span class="nx">query</span><span class="p">.</span><span class="nx">Password</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">err</span> <span class="o">:=</span> <span class="nx">loginUsingGrafanaDB</span><span class="p">(</span><span class="nx">query</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">||</span> <span class="p">(</span><span class="nx">err</span> <span class="o">!=</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrUserNotFound</span> <span class="o">&amp;&amp;</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">ErrInvalidCredentials</span> <span class="o">&amp;&amp;</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">ErrUserDisabled</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">ldapEnabled</span><span class="p">,</span> <span class="nx">ldapErr</span> <span class="o">:=</span> <span class="nx">loginUsingLDAP</span><span class="p">(</span><span class="nx">query</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">ldapEnabled</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">ldapErr</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">||</span> <span class="nx">ldapErr</span> <span class="o">!=</span> <span class="nx">ldap</span><span class="p">.</span><span class="nx">ErrInvalidCredentials</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">ldapErr</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">ErrUserDisabled</span> <span class="o">||</span> <span class="nx">ldapErr</span> <span class="o">!=</span> <span class="nx">ldap</span><span class="p">.</span><span class="nx">ErrInvalidCredentials</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">err</span> <span class="p">=</span> <span class="nx">ldapErr</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">ErrInvalidCredentials</span> <span class="o">||</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">ldap</span><span class="p">.</span><span class="nx">ErrInvalidCredentials</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">saveInvalidLoginAttempt</span><span class="p">(</span><span class="nx">query</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">loginLogger</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="s">&quot;Failed to save invalid login attempt&quot;</span><span class="p">,</span> <span class="s">&quot;err&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">ErrInvalidCredentials</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrUserNotFound</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">ErrInvalidCredentials</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>总线机制是如何工作的？</h3>

<p>到处都看到<code>bus.Dispatch</code>，那么bus是如何知道action对应的handler的呢？一定有一个register的机制，但没找到呢</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 这样注册</span>
</span><span class='line'><span class="nx">bus</span><span class="p">.</span><span class="nx">AddHandler</span><span class="p">(</span><span class="s">&quot;sql&quot;</span><span class="p">,</span> <span class="nx">UpdateUserLastSeenAt</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 这里是handler的实现</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">UpdateUserLastSeenAt</span><span class="p">(</span><span class="nx">cmd</span> <span class="o">*</span><span class="nx">models</span><span class="p">.</span><span class="nx">UpdateUserLastSeenAtCommand</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">inTransaction</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">sess</span> <span class="o">*</span><span class="nx">DBSession</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">user</span> <span class="o">:=</span> <span class="nx">models</span><span class="p">.</span><span class="nx">User</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">Id</span><span class="p">:</span>         <span class="nx">cmd</span><span class="p">.</span><span class="nx">UserId</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">LastSeenAt</span><span class="p">:</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">(),</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">sess</span><span class="p">.</span><span class="nx">ID</span><span class="p">(</span><span class="nx">cmd</span><span class="p">.</span><span class="nx">UserId</span><span class="p">).</span><span class="nx">Update</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">user</span><span class="p">)</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">})</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// AddHandler的实现</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">InProcBus</span><span class="p">)</span> <span class="nx">AddHandler</span><span class="p">(</span><span class="nx">handler</span> <span class="nx">HandlerFunc</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">handlerType</span> <span class="o">:=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">TypeOf</span><span class="p">(</span><span class="nx">handler</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">queryTypeName</span> <span class="o">:=</span> <span class="nx">handlerType</span><span class="p">.</span><span class="nx">In</span><span class="p">(</span><span class="mi">0</span><span class="p">).</span><span class="nx">Elem</span><span class="p">().</span><span class="nx">Name</span><span class="p">()</span>
</span><span class='line'>  <span class="c1">// 原来所谓注册就是一个map，key是handler的入参，val是handler</span>
</span><span class='line'>  <span class="nx">b</span><span class="p">.</span><span class="nx">handlers</span><span class="p">[</span><span class="nx">queryTypeName</span><span class="p">]</span> <span class="p">=</span> <span class="nx">handler</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Dispatch的实现</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">Dispatch</span><span class="p">(</span><span class="nx">msg</span> <span class="nx">Msg</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">globalBus</span><span class="p">.</span><span class="nx">Dispatch</span><span class="p">(</span><span class="nx">msg</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">InProcBus</span><span class="p">)</span> <span class="nx">DispatchCtx</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">msg</span> <span class="nx">Msg</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">msgName</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">TypeOf</span><span class="p">(</span><span class="nx">msg</span><span class="p">).</span><span class="nx">Elem</span><span class="p">().</span><span class="nx">Name</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">handler</span> <span class="p">=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">handlersWithCtx</span><span class="p">[</span><span class="nx">msgName</span><span class="p">]</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">handler</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">ErrHandlerNotFound</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">params</span> <span class="p">=</span> <span class="p">[]</span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Value</span><span class="p">{}</span>
</span><span class='line'>  <span class="nx">params</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">params</span><span class="p">,</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">ctx</span><span class="p">))</span>
</span><span class='line'>  <span class="nx">params</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">params</span><span class="p">,</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">msg</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// 反射来调用</span>
</span><span class='line'>  <span class="nx">ret</span> <span class="o">:=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">handler</span><span class="p">).</span><span class="nx">Call</span><span class="p">(</span><span class="nx">params</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ret</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">Interface</span><span class="p">()</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">err</span><span class="p">.(</span><span class="kt">error</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>setting是如何初始化的？</h3>

<p>看到使用的时候是直接这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">if</span> <span class="p">!</span><span class="nx">setting</span><span class="p">.</span><span class="nx">BasicAuthEnabled</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>dto的运用</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// pkg/api/dashboard.go</span>
</span><span class='line'>      <span class="nx">dashItem</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">dashboards</span><span class="p">.</span><span class="nx">SaveDashboardDTO</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">Dashboard</span><span class="p">:</span> <span class="nx">dash</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">Message</span><span class="p">:</span>   <span class="nx">cmd</span><span class="p">.</span><span class="nx">Message</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">OrgId</span><span class="p">:</span>     <span class="nx">c</span><span class="p">.</span><span class="nx">OrgId</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">User</span><span class="p">:</span>      <span class="nx">c</span><span class="p">.</span><span class="nx">SignedInUser</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">Overwrite</span><span class="p">:</span> <span class="nx">cmd</span><span class="p">.</span><span class="nx">Overwrite</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">dashboard</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">dashboards</span><span class="p">.</span><span class="nx">NewService</span><span class="p">().</span><span class="nx">SaveDashboard</span><span class="p">(</span><span class="nx">dashItem</span><span class="p">,</span> <span class="nx">allowUiUpdate</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">dashboardSaveErrorToApiResponse</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>业务error到api error的转换</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="nx">dashboard</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">dashboards</span><span class="p">.</span><span class="nx">NewService</span><span class="p">().</span><span class="nx">SaveDashboard</span><span class="p">(</span><span class="nx">dashItem</span><span class="p">,</span> <span class="nx">allowUiUpdate</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">dashboardSaveErrorToApiResponse</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">dashboardSaveErrorToApiResponse</span><span class="p">(</span><span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="nx">Response</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardTitleEmpty</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardWithSameNameAsFolder</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardFolderWithSameNameAsDashboard</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardTypeMismatch</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardInvalidUid</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardUidToLong</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardWithSameUIDExists</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrFolderNotFound</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardFolderCannotHaveParent</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardFolderNameExists</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardRefreshIntervalTooShort</span> <span class="o">||</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardCannotSaveProvisionedDashboard</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">Error</span><span class="p">(</span><span class="mi">400</span><span class="p">,</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span> <span class="kc">nil</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardUpdateAccessDenied</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">Error</span><span class="p">(</span><span class="mi">403</span><span class="p">,</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">validationErr</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">alerting</span><span class="p">.</span><span class="nx">ValidationError</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">Error</span><span class="p">(</span><span class="mi">422</span><span class="p">,</span> <span class="nx">validationErr</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span> <span class="kc">nil</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardWithSameNameInFolderExists</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">JSON</span><span class="p">(</span><span class="mi">412</span><span class="p">,</span> <span class="nx">util</span><span class="p">.</span><span class="nx">DynMap</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;name-exists&quot;</span><span class="p">,</span> <span class="s">&quot;message&quot;</span><span class="p">:</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">()})</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardVersionMismatch</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">JSON</span><span class="p">(</span><span class="mi">412</span><span class="p">,</span> <span class="nx">util</span><span class="p">.</span><span class="nx">DynMap</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;version-mismatch&quot;</span><span class="p">,</span> <span class="s">&quot;message&quot;</span><span class="p">:</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">()})</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">pluginErr</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">models</span><span class="p">.</span><span class="nx">UpdatePluginDashboardError</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">message</span> <span class="o">:=</span> <span class="s">&quot;The dashboard belongs to plugin &quot;</span> <span class="o">+</span> <span class="nx">pluginErr</span><span class="p">.</span><span class="nx">PluginId</span> <span class="o">+</span> <span class="s">&quot;.&quot;</span>
</span><span class='line'>      <span class="c1">// look up plugin name</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">pluginDef</span><span class="p">,</span> <span class="nx">exist</span> <span class="o">:=</span> <span class="nx">plugins</span><span class="p">.</span><span class="nx">Plugins</span><span class="p">[</span><span class="nx">pluginErr</span><span class="p">.</span><span class="nx">PluginId</span><span class="p">];</span> <span class="nx">exist</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">message</span> <span class="p">=</span> <span class="s">&quot;The dashboard belongs to plugin &quot;</span> <span class="o">+</span> <span class="nx">pluginDef</span><span class="p">.</span><span class="nx">Name</span> <span class="o">+</span> <span class="s">&quot;.&quot;</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">JSON</span><span class="p">(</span><span class="mi">412</span><span class="p">,</span> <span class="nx">util</span><span class="p">.</span><span class="nx">DynMap</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;plugin-dashboard&quot;</span><span class="p">,</span> <span class="s">&quot;message&quot;</span><span class="p">:</span> <span class="nx">message</span><span class="p">})</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">models</span><span class="p">.</span><span class="nx">ErrDashboardNotFound</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">JSON</span><span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="nx">util</span><span class="p">.</span><span class="nx">DynMap</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;not-found&quot;</span><span class="p">,</span> <span class="s">&quot;message&quot;</span><span class="p">:</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">()})</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">return</span> <span class="nx">Error</span><span class="p">(</span><span class="mi">500</span><span class="p">,</span> <span class="s">&quot;Failed to save dashboard&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[极客时间「软件工程之美」学习笔记]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-geektime-software-engineering/"/>
    <updated>2021-02-05T11:58:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-geektime-software-engineering</id>
    <content type="html"><![CDATA[<p><a href="https://time.geekbang.org/column/article/82337">软件工程之美</a></p>

<p>配套书籍：构建之法</p>

<p>[toc]</p>

<h3>开篇</h3>

<p>未来10年，什么不会变？</p>

<blockquote><p>数据结构、算法、面向对象思想、设计模式、软件工程</p></blockquote>

<h3>基础理论</h3>

<h4>怎样理解软件工程</h4>

<p>像这种有人参与、有计划、有步骤地造一件产品，我们通常称为“工程”。</p>

<p>就是要用工程化方法去规范软件开发，让项目可以按时完成、成本可控、质量有保证。</p>

<h4>工程思维</h4>

<p>软件工程不仅可以应用在软件项目中，还可以应用于日常生活中遇到的一些问题，Everything is a project</p>

<p>站在整体而非局部去看问题</p>

<p>工程思维，本质上是一种思考问题的方式，在解决日常遇到的问题时，尝试从一个项目的角度去看待问题、尝试用工程方法去解决问题、站在一个整体而不是局部的角度去看问题。</p>

<p><a href="https://zhuanlan.zhihu.com/p/21314651">记录下两个孩子在MineCraft里面还原公寓的经历</a></p>

<h4>瀑布模型</h4>

<p>问题的定义与规划</p>

<p>需求分析</p>

<p>软件设计</p>

<p>程序编码</p>

<p>软件测试</p>

<p>运行维护</p>

<p>优缺点：阶段分的很清楚，但有点死板了。对于前期需求不明确的项目，很难开展需求分析，后续如果有需求变更，瀑布模型便很难响应</p>

<h4>其它模型</h4>

<h5>快速原型模型</h5>

<p>先迅速建造一个可以运行的软件原型，然后收集用户反馈，再反复修改确认，使开发出的软件能真正反映用户需求，这种开发模型就叫快速原型模型，也叫原型模型</p>

<h5>增量模型</h5>

<p>按模块分批次交付</p>

<h5>迭代模型</h5>

<p>每次迭代都是一个可用的版本</p>

<h4>敏捷开发</h4>

<p>敏捷不是一种方法论，也不是一种软件开发的具体方法，更不是一个框架或过程，<strong>而是一套价值观和原则</strong>。</p>

<p>当你开发做决策的时候，遵守了敏捷开发的价值观和原则，不管你是不是用 Scrum 或者极限编程，那么都可以算是敏捷开发。</p>

<h4>项目管理金三角</h4>

<p>做产品想“多、快、好、省”都占着，是不可能的，最多只能选两样。</p>

<p>想要便宜和质量好，就要花时间等；想要快还要质量好，那就得多花钱；想要又便宜又快，那就得接受难用、质量差。</p>

<p>在软件项目中，有一个类似的平衡关系，就是软件质量（产品的质量，客户的满意度）与范围（需要实现多少功能）、时间（多久可以完成）、成本（花多少钱）四个要素之间的平衡。</p>

<h3>项目规划篇</h3>

<h4>可行性分析</h4>

<p>如何科学地论证项目的可行性，以及这个项目是不是值得做</p>

<p>可分为经济可行性、技术可行性、社会可行性</p>

<h4>如果你想技术转管理，先来试试管好一个项目</h4>

<h4>代码未动，计划先行</h4>

<p>如果没有计划，你的项目可能会陷入一种无序和混乱中。</p>

<p>参与做计划的过程，可以让你对项目的各种事情了然于胸，这就相当于扩大了你的上下文，让你有更高的视角看待当前工作遇到的问题。</p>

<blockquote><p>如何制定计划？</p></blockquote>

<p>第一步：任务分解；</p>

<p>意思是工作分解结构（Work Breakdown Structure, WBS)。就是把要做的事情，按照一个树形结构去组织，逐级分解，分割成小而具体的可交付结果，直到不能再拆分为止</p>

<p>第二步：估算时间；</p>

<p>主要还是得依靠以前的经验:</p>

<ul>
<li><p>任务拆分的越细致，想的越清楚，就能估算的越准确</p></li>
<li><p>要让负责这个任务的人员参与估算</p></li>
</ul>


<p>在沟通中也要注意技巧，<strong>不要采用质问的方式</strong>：“这么简单一个模块居然要 5 天？”这只会让听者产生逆反心理，无法有效的沟通。可以恰当的提一些问题来达到有效沟通的目的，比如我通常会问两个问题：</p>

<ul>
<li>“能不能把你这个任务再细化一下？”</li>
<li>“能不能简单介绍一下这个模块你是打算如何实现的？”</li>
</ul>


<p>对于估算的结果，<strong>通常还要考虑增加一些余量</strong>，因为实际项目执行过程中，并没办法保证是 100% 投入，有可能并行还有其他事情，或者一些突发事情、事先没有考虑到的任务都有可能影响进度。</p>

<p>第三步：排任务路径。</p>

<p>项目中有些任务是可以并行做的，而有些任务之间则是有依赖关系的。</p>

<p>排路径就是要根据任务之间的关系，资源的占用情况，排出合适的顺序。</p>

<p><strong>制定计划时不要担心不够准确，先有一个基本的计划，可以粒度比较粗，不那么准确，让事情先推进起来。</strong></p>

<blockquote><p>设置里程碑</p></blockquote>

<p>周期很长的项目，一直看不到结果，时间一长会很疲惫。所以有经验的项目经理会在项目启动后，根据制订好的初步计划，确定几个关键的里程碑</p>

<p>里程碑代表着一份承诺，在里程碑完成后，大家会获得一种正面激励。</p>

<p>在项目的推进过程中，根据里程碑完成的情况，你就可以很直观地知道项目的进展如何。如果发现不能如期完成里程碑，就需要进行适当的调整了</p>

<blockquote><p>计划需要跟踪和调整</p></blockquote>

<p>跟踪进度的方式主要有两种，一种是项目经理定期收集跟踪，一种是项目成员主动汇报。项目经理挨个收集的话，会有一个沟通确认的过程，对进度会了解的更准确；项目成员主动汇报，可以减少项目经理的收集工作，但有可能不准确。</p>

<h4>流程和规范：红绿灯不是约束，而是用来提高效率</h4>

<p>从个体来看，因为流程规范的存在，确实可能存在效率降低的情况，但从团队的角度来看，好的流程规范反而是提升效率的。</p>

<p>以代码审查的规范为例，对于技术高的程序员来说，代码审查可能会耽误一点时间，但对整个团队来讲：</p>

<ul>
<li>即使是水平高的程序员，也可能会被发现有错误，代码审查可以降低出错的概率，保障质量</li>
<li>对于水平低的程序员，可以通过代码审查学习和成长，代码被高水平程序员审查后，可以有效提高质量。</li>
</ul>


<blockquote><p>如何制定好流程规范？</p></blockquote>

<p>第一步：明确要解决的问题</p>

<p>要制定一个流程规范，第一步就是明确你是要解决什么样的问题。项目中很多问题，都可以思考是不是能通过流程解决。</p>

<p>第二步：提出解决方案</p>

<p>第三步：达成共识，推广执行</p>

<p>在流程规范提出后，还需要得到大家认可，只有大家认可，达成共识，才能共同遵守，保障制度的执行。</p>

<p>第四步： 持续优化，不断改进</p>

<p>流程制定后，在实际执行的时候，难免发现一些不合理或者不科学的地方，这时候就需要对其进行调整。</p>

<blockquote><p>将流程规范工具化</p></blockquote>

<p>尽可能借助技术手段来推动甚至替代流程规范。</p>

<h4>白天开会，加班写代码的节奏怎么破？</h4>

<blockquote><p>如何提高开会效率？</p></blockquote>

<p>我们先要认识到一个前提：<strong>那就是要让大家意识到开会是有成本的，如果开会创造的价值不能大于其成本，就是浪费。</strong></p>

<ol>
<li>砍掉一些没价值的会议</li>
<li>减少参与会议的人。会议的成本和两个因素相关：一个是人数，一个是时间。如果减少人数，就能减少成本。</li>
<li>缩短开会时间</li>
<li>提升会议所创造的价值</li>
</ol>


<h4>项目管理工具：一切管理问题，都应思考能否通过工具解决</h4>

<blockquote><p>基于 Ticket 的任务跟踪系统</p></blockquote>

<p>那一个 Ticket 应该包含哪些主要信息呢？</p>

<ul>
<li>标题：摘要性的描述 Ticket 内容；</li>
<li>类型：属于什么类型的 Ticket：Bug、需求、任务；</li>
<li>内容：Ticket 的详细内容，例如，如果是 Bug 的话，除了要写清楚 Bug 内容，还需要重现步骤。如果是需求的话，要有需求的描述，可能还需要额外的文档链接辅助说明；</li>
<li>创建人：谁创建的这条 Ticket；</li>
<li>优先级：这个 Ticket 的优先级高还是低；</li>
<li>状态：Ticket 的状态，例如：未开始、处理中、已解决、重新打开、关闭等；</li>
<li>指派给谁：这个 Ticket 被指派给谁了，谁来负责；</li>
<li>历史记录：整个 Ticket 改变的历史信息，用以跟踪；</li>
</ul>


<h4>风险管理：不能盲目乐观，凡事都应该有B计划</h4>

<p>风险管理就是指在项目进行过程中，识别可能的风险，对风险进行评估，并加以监控，从而减少风险对项目的负面影响。</p>

<blockquote><p>如何做好风险管理？</p></blockquote>

<p>培养风险意识</p>

<p>项目中的任务，不能盲目乐观，都思考一下它最坏的结果是什么，如果最坏的结果不能接受，就说明要有个 B 计划，考虑风险管理了。</p>

<p>管理风险</p>

<p>第一步：风险识别，识别可能的风险</p>

<ul>
<li>项目风险：项目预算、进度、用户和需求等方面的问题；</li>
<li>人员风险：人员离职、人手不足等问题；</li>
<li>技术风险：采用的技术所可能带来的风险；</li>
<li>商业风险：与市场、产品策略等有关的商业风险。</li>
</ul>


<p>第二步：风险量化，对风险进行评估量化</p>

<p>第三步：应对计划，对风险制定应对策略</p>

<p>第四步：风险监控，对风险进行监控预警</p>

<h4>怎样才能写好项目文档？</h4>

<blockquote><p>为什么要写文档？</p></blockquote>

<p>写文档，其实对个人、对项目、对团队，都是非常重要的事情。</p>

<ul>
<li>帮助写文档的人理清楚思路.先写文档。先写文档，就会抛开代码细节，去站在全局思考</li>
</ul>


<p>有人觉得自己写作不行，所以不会写文档。写作不行，只是让你在用词遣句上会有所欠缺，而这不是写文档的真正障碍</p>

<p><strong>真正的障碍是没想清楚，在心中只有一些未成型的混乱的想法和概念，必须要努力把这些模糊的想法确定化和具体化，才能写出来。</strong></p>

<p>换个角度来说，如果你连文档都写不出来，那又怎么能指望代码写得好呢？</p>

<ul>
<li>便于未来的维护和交接</li>
</ul>


<p>我有一个习惯，每到一个新项目组，就会把日常工作中遇到的问题、各种环境配置、一些操作的步骤等，所有以后可能还会用上的都记录下来，其中一些还会整理到团队的 WIKI 上</p>

<ul>
<li>团队更好的协作沟通</li>
</ul>


<blockquote><p>如何写好文档？</p></blockquote>

<p>很多人对于写文档是有心理压力，觉得自己写作水平不高，不知道该如何下手。首先你要对文档有一个正确的认识：文档写作，<strong>关键是通过文档把你的想法表达出来</strong>，至于用词、格式相对都是其次的。</p>

<ul>
<li>从模仿开始</li>
<li>从小文档开始</li>
</ul>


<p>项目中很多文档都可以从这样小的内容开始：别人给你讲一个问题的时候记录下来；你给别人讲一个问题的时候记录下来；解决一个技术难题时记录下来方案……</p>

<p>这些记录下来的笔记，稍加整理，就可以是很不错的项目文档。</p>

<ul>
<li>从粗到细，迭代更新</li>
</ul>


<p>为什么我不一开始就写很细的文档呢？</p>

<p>一个原因是太难写，要花很多时间精力，甚至可能写不下去；另一个原因就是在收集反馈的过程中，会有很多修改。<strong>写得越细则无用功越多，最后，你甚至会因为不想改文档而抵触不同的意见。</strong></p>

<blockquote><p>基本的画图的技巧</p></blockquote>

<p>写文档的时候，主要有几种图比较常用：线框图、流程图、时序图、各种格式的截图。</p>

<ul>
<li>线框图</li>
<li>流程图</li>
<li>时序图</li>
</ul>


<p>在评审技术文档或者你自己写技术文档，你可以从几个角度去思考：</p>

<ol>
<li><p>文档是否讲清楚了它的目的是什么？内容是否和目的匹配？
比如说你这个设计文档是要解决什么问题？设计的是哪个模块？</p></li>
<li><p>文档是否解释了为什么要这么做？
比如说你这个模块设计文档，是否解释清楚了为什么要这样设计？这样设计的优缺点是什么？</p></li>
<li><p>文档是否描述清楚了如何实现？
比如说作为模块设计文档，到底是如何设计的？有没有讲清楚整体的设计架构？</p></li>
</ol>


<p>总结一下就是一个技术文档，要讲清楚：是什么（What）？为什么（Why）？怎么做（How）？这三个是最核心的要素。</p>

<p>这三个核心问题讲清楚了，然后就是多画图，内容简洁明了。</p>

<blockquote><p>例子？</p></blockquote>

<p><a href="https://docshome.gitbooks.io/microservices/content/2-using-an-api-gateway.html">微服务：从设计到部署</a></p>

<h3>需求分析篇</h3>

<h3>系统设计篇</h3>

<h4>架构设计：普通程序员也能实现复杂系统？</h4>

<p>软件需求越来越多，而高手又是稀缺资源，所以要解决的一个问题就是：让普通程序员也能参与其中，一起实现复杂系统，而不必依赖于很多精英。</p>

<blockquote><p>为什么软件项目需要架构设计？</p></blockquote>

<p>复杂的软件项目，通常有两个特点：需求不确定和技术复杂。</p>

<blockquote><p>什么是架构设计？</p></blockquote>

<p>最开始我以为架构设计的目标是满足业务需求，保证软件能正常工作。后来发现<strong>这其实只是最基本的要求，因为很多糟糕的架构设计，也能满足业务需求，让系统正常运行。</strong></p>

<p>架构设计的目标，是<strong>用最小的人力成本来满足需求的开发和响应需求的变化，用最小的运行成本来保障软件的运行。</strong></p>

<p>架构设计，已经有很多成熟的方法:</p>

<ul>
<li>使用微服务这样的架构，把复杂系统拆分成一系列小的服务，服务再拆成功能模块，让人员更好地分工协作；</li>
<li>通过前后端分离，让程序员更专注于某个知识领域，降低开发难度；</li>
<li>用分层设计来隔离业务逻辑，减少需求变更带来的影响。</li>
</ul>


<p>这些架构设计的方法，其实都是基于工程领域分而治之的策略，本质上就是将系统分拆，将人员分拆。<strong>但是光拆还不够，拆完了还得能拼回来</strong></p>

<blockquote><p>如何做好架构设计？</p></blockquote>

<p>第一步：分析需求</p>

<p>架构设计，最基本的就是要能满足业务需求，所以搞清楚需求是至关重要一步。<strong>而产品需求，只有功能的描述，界面的交互，还需要进一步进行抽象。</strong></p>

<p>一个常用的分析方法就是分析用例，也就是了解主要用户角色和其使用的场景。</p>

<p>第二步：选择相似的成熟的架构设计方案</p>

<p><a href="http://www.ruanyifeng.com/blog/2016/09/software-architecture.html">软件架构入门</a></p>

<p><a href="https://yq.aliyun.com/articles/57715">异地多活设计辣么难？其实是你想多了！</a></p>

<p>这篇文章有醍醐灌顶的感觉~  解开了不少疑惑！</p>

<p>第三步：自顶向下层层细化</p>

<p>第四步：验证和优化架构设计方案</p>

<blockquote><p>推荐材料</p></blockquote>

<ul>
<li><a href="https://www.w3cschool.cn/architectroad/">架构师之路</a></li>
<li><a href="https://mp.weixin.qq.com/s/wSCeO8QVYniMIGcBFDZyjw?">图解：从单个服务器扩展到百万用户的系统</a></li>
<li><a href="https://www.infoq.cn/article/architecture-design-process">以“前浪微博”场景为例，谈谈架构设计流程四步曲</a></li>
<li><a href="https://book.douban.com/subject/30333919/">架构整洁之道</a></li>
<li><a href="https://www.infoq.cn/article/q7xlHaaiZ-9H6SwHXCNg">极客邦池建强：难的不是从零打造一款产品，而是……</a></li>
<li><a href="https://book.douban.com/subject/1761742/">Head First Object-Oriented Analysis and Design</a></li>
<li><a href="http://www.ruanyifeng.com/blog/2016/09/software-architecture.html">软件架构入门</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[「影响力」读书笔记]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-influence/"/>
    <updated>2021-02-05T11:54:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/02/05/reading-notes-of-influence</id>
    <content type="html"><![CDATA[<h3>互惠</h3>

<p>一个古老的原理：给予、索取。。。再索取</p>

<p>互惠原因认为，我们应该尽量以相同的方式回报他人为我们所做的一切</p>

<p>当前社会的普遍价值观认为「互惠」是一种好的行为，若违背了它会遭到无情的唾弃和嘲弄，甚至会被戴上乞讨赖账、忘恩负义之类的大帽子。我们为了维护自己的形象，往往会努力遵循这个原则，但也给了一些人利用这个原理获利的机会</p>

<p>互惠原理的几个重要特点：</p>

<ul>
<li>互惠原理具有压倒性的力量</li>
</ul>


<p>意味着即使是我们不喜欢的人（不请自来的推销员、令人讨厌的点头之交等），只要在提出要求之前送我们一个小小的人情，我们对他们的要求就失去了抵抗力。</p>

<p>例子：</p>

<p>问卷调查时附带一些钱作为礼物，会明显提供问卷的答复率（对比，回答了问卷后，再给奖励，效果并不好）</p>

<p>生意人发现，在收到一件礼物后，顾客们有可能会购买他们本来会拒绝的服务或商品</p>

<p>免费试用（据说目的是让他们试一下看自己到底喜不喜欢这个产品），但实际上免费试用品也是一种礼品。比如，超市中的试吃，你吃了后还会那么从容的走开吗？</p>

<p>吉姆 琼斯：集体自杀事件中，大部分人都顺从的服毒自杀，但没有自杀的人中，拒绝自杀的命令是因为“当初在有需要的时候拒绝了吉姆 琼斯的帮助”</p>

<ul>
<li>强加于人的负债感</li>
</ul>


<p>互惠原理只是说我们应该回报人家的善举，而并没有说只有当我们主动要求了这个善举时才有回报的义务。当这个好处是不请自来时，这种负债的感觉也是照样存在的</p>

<h4>「互惠」的变种</h4>

<ul>
<li>相互退让</li>
</ul>


<p>如果他人对我们作出让步，我们也有义务作出让步</p>

<ul>
<li>拒绝 &ndash; 退让策略</li>
</ul>


<p>先提出一个比较大的、极可能被拒绝的请求，然后在被拒绝后，再提出你真正想提的请求</p>

<p>例子：</p>

<p>如果你要向别人推销，就要先展示质次价高的商品；如果你要向别人借钱，就要先开口借一个很高的金额。这样会增加第二个请求被接受的可能性</p>

<h4>如何保护自己</h4>

<p>好像也没有很好的办法</p>

<ul>
<li>拒绝好意？ 有可能会误伤</li>
<li>有意识的提醒自己</li>
</ul>


<h3>承诺和一致</h3>

<p>如果一开始没拒绝，后来就难了</p>

<p>我们都有一种要做到与过去的行为相一致的愿望。一旦我们作出了一个觉得或者选择了一种立场，就会有发自内心以及来自外部的压力来迫使我们与此保持一致。在这种压力下，我们总是希望以实际行动来证明我们以前的决定是正确的</p>

<p>例子：</p>

<p>一投完票，选民们往往就会更相信他们的候选人一定会当选</p>

<p>玩具商先鼓吹一个玩具，当又不供货，家长买不到只能先买别的替代品，然后玩具商又鼓吹一波，这次有货了，为了信守承诺，家长又买了一次</p>

<h4>为什么会这样？</h4>

<p>很多情况下，保持一致是一种很有益的行为；始终如一的个性是受到高度评价的</p>

<h4>如何被利用</h4>

<p>先引诱我们采取某种行动或作出某种声明，然后再利用我们要与过去保持一致的压力来迫使我们屈从于他们的要求</p>

<p>例子：</p>

<p>音乐会广告隐藏票价（即使是一个电话，也是对参加音乐会的一种初步承诺）</p>

<p>先给出一个很有诱惑力的价格，使顾客作出买车的决定，然后当这个决定已经作出但还没有最后成交时，那个最先给出的诱惑却被巧妙的拿走了</p>

<h4>如何保护自己</h4>

<p>唯一有效的办法是对原理有一个清醒的认识。虽然保持一致性通常来说是应该的，甚至是至关重要的，但愚蠢地、机械地保持一致却是应该避免的</p>

<h3>社会认同</h3>

<p>在大家都用同样的方式思考的地方，没有人思考的很深刻</p>

<p>为什么有些电视中会使用配音笑声？为什么笑声明明是假的，却仍然可以左右我们的行动？</p>

<p>社会认同原理认为，我们进行是非判断的标准之一就是看别人是怎么想的，尤其是当我们要决定什么是正确的行为时。如果我们看到别人在某种场合做某件事情，我们就会断定这样做是有道理的。我们周围的人的做法对我们决定自己应该怎么行动都有重要的指导意义。</p>

<p>例子：</p>

<p>小费盘中先放几块钱，假装是前面的顾客留下的</p>

<p>广告商最喜欢告诉我们某种商品的增长最快或者销售最旺，不必说他们的产品很优良，而只需说其他人都这样认为，这个证据已经很充分了</p>

<p>慈善节目里主持人会花大量时间念一长串已经捐赠的观众名单</p>

<p>夜总会老板在舞厅还有空余的情况下故意让顾客在外面排长龙（让大家觉得，这么多人都去了。。）</p>

<blockquote><p>旁观者效应</p></blockquote>

<ul>
<li>别人不救，我也不救</li>
</ul>


<p><a href="https://zh.wikipedia.org/wiki/%E5%B0%8F%E6%82%A6%E6%82%A6%E4%BA%8B%E4%BB%B6">小悦悦事件</a>：2岁女童王悦（乳名“小悦悦”）在阴雨天黑夜独自跑出家门百米外后，先后被两辆汽车撞伤倒地，最初路过的18名行人未及时施救，惟第19名路人陈贤妹（生于1953年，佛山本地人，职业是清洁工）救起王悦</p>

<p>如何破解？  明确指定一个人来寻求帮助</p>

<ul>
<li>别人进，我也进</li>
</ul>


<p>股票牛市大家都冲进去</p>

<h4>影响因子</h4>

<ul>
<li>不确定性</li>
</ul>


<p>当人们对自己的处境不是很有把握时，更有可能根据他人的行为来决定自己应该怎么办</p>

<ul>
<li>相似性</li>
</ul>


<p>我们更有可能效仿与我们相同而不是不同的人的行动</p>

<h4>如何保护自己</h4>

<p>大部分情况下，这个原理是非常有用处的</p>

<p>困境：如何利用一种既对我们有利又对我们有害的装置</p>

<ul>
<li>识别被伪造的社会证据。比如前面说的假笑、广告里的普通人</li>
<li>多观察、多思考、多求证</li>
</ul>


<h3>喜好</h3>

<p>人们总是愿意答应自己认识和喜爱的人提出的要求，这应该是很自然的。但这条原理却被一些想要我们答应他们要求的陌生人利用了</p>

<ul>
<li>外表吸引力的光环</li>
</ul>


<p>例子：</p>

<p>骗子通常都是俊男靓女</p>

<ul>
<li>相似性</li>
</ul>


<p>人都喜欢跟自己类似的人，不管是什么方面，所谓的聊的来</p>

<ul>
<li>称赞</li>
</ul>


<h4>如何保护自己</h4>

<p>时刻自问：是否对「对方」过于喜爱了？</p>

<p>提醒自己，把交易者和交易分开</p>

<h3>权威</h3>

<p>跟着行家走</p>

<p>遵从权威对我们来说，通常都是一件很实际的事情。比如，年幼的时候，老师和父母是我们心目中的权威人物，我们发现听他们的话是很明智的。这一方面是因为他们拥有更多的智慧，另一方面是由于他们就是决定奖赏的人</p>

<p>成年后，因为同样的原因，我们还是愿意接受权威的忠告，只不过现在的权威人物变成了雇主、法官和政府。因为他们所处的有利位置，他们可以接触更多的信息，拥有更多的权力，因此按照他们的意志行事大体上是错不了的。</p>

<p>而正是由于我们对这一点深信不疑，也很容易走进一个误区：有时候权威的话并没有什么道理，可是我们还是会毫不犹豫的按照他们所说的做</p>

<p>例子：</p>

<p>名人做广告。扮演皇帝的演员</p>

<ul>
<li>头衔</li>
</ul>


<p>骗子喜欢用头衔、衣着和象征身份地位的外部标志</p>

<ul>
<li>衣着</li>
</ul>


<p>假警察</p>

<p>网络诈骗，一般会声称自己是公安局的</p>

<ul>
<li>外部标志</li>
</ul>


<p>有豪车</p>

<h4>如何保护自己</h4>

<p>做好充分的思想准备，对权威保持高度的警觉。有了这种警觉，同时也意识到权威是多么容易假冒，在遇到有人想利用权威来影响我们的场合，我们自然就会采取一种比较谨慎的态度</p>

<h3>短缺</h3>

<p>物以稀为贵</p>

<ul>
<li>数量有限</li>
</ul>


<p>例子：</p>

<p>告诉顾客某种产品数量有限，不能保证一直有货。&mdash;&mdash;  饥饿营销</p>

<ul>
<li>时间限制</li>
</ul>


<p>截止时间策略</p>

<p>例子：</p>

<p>审查制度使得“禁片”更加流行，删减片段更容易传播</p>

<h4>如何保护自己</h4>

<p>面临短缺的压力时，问自己：我们为什么想要这件东西？如果我们的回答是主要目的是拥有它，那么我们就应该根据它的稀缺程度来决定它的价格。然而，如果回答是我们之所以想要它，主要是为了它的功能（驾驶、吃、穿、喝的东西），那么我们就应该记住，不管这种东西的供应是否充足，它的功能是一样的。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Golang工程最佳实践]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/01/26/golang-best-practice/"/>
    <updated>2021-01-26T18:20:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/01/26/golang-best-practice</id>
    <content type="html"><![CDATA[<h3>examples</h3>

<ul>
<li><a href="https://semaphoreci.com/community/tutorials/building-go-web-applications-and-microservices-using-gin">Building Go Web Applications and Microservices Using Gin</a> 有实现，有测试，有重构</li>
</ul>


<h3>项目分层</h3>

<ul>
<li>model

<ul>
<li>所有的数据模型，被其它层所依赖</li>
</ul>
</li>
<li>repo

<ul>
<li>数据层，屏蔽所有数据存取的细节</li>
<li>它的底层可以是数据库，也可以是<strong>其它服务</strong></li>
</ul>
</li>
<li>business

<ul>
<li>业务逻辑层，依赖数据层</li>
</ul>
</li>
<li>controller

<ul>
<li>负责参数输入的验证、统一返回结果，依赖业务逻辑层</li>
</ul>
</li>
</ul>


<p>思考：</p>

<p>理想情况下，一个项目应该有上面几层，但又想到，其实应该按照一个项目的复杂度来决定它的分层，当项目的复杂度不高时，非要分成上述这么多层，反而会比较繁琐（毕竟，逻辑写在一个文件里最清楚了。。）。</p>

<p>还有一个问题值得思考，就是我们是否应该将每层都定义成接口，然后层与层之间只是接口依赖，不依赖于具体的实现？这样做，</p>

<ul>
<li>好处：

<ul>
<li>更灵活，每一层的实现都是可随便替换的；</li>
<li>更容易写单元测试，基于接口可以很容易写一个测试专用的mock实现。</li>
</ul>
</li>
<li>坏处：

<ul>
<li>如果不需要写单元测试（写针对接口的集成测试），上面的好处也没多大意义了</li>
<li>如果实现（比如MySQL &mdash;> PG）基本不可能换，上面的好处也没多大意义</li>
</ul>
</li>
</ul>


<p>单纯从易于测试这个问题来说，基于接口的集成测试所需的时间肯定比mock依赖的单元测试要长，但感觉目前遇到的项目的复杂度并不高，执行一次集成测试的时间也还可以接受，因此暂且仅仅使用针对接口的集成测试看看效果，后续如果感觉需要加单元测试再慢慢迭代吧，随着认知的成长，越来越认识到，任何事物（项目、个人、公司等）的发展都有一个过程，迭代是个美妙的词，我不应该苛求每次工作刚开始做就要达到一个很高的标准，应该慢慢迭代，第一期达到基本可用，第二期增加xx功能，等等等等。</p>

<p>参考：</p>

<ul>
<li><a href="https://deliveroo.engineering/2019/05/17/testing-go-services-using-interfaces.html">Testing Go services using interfaces</a></li>
<li><a href="https://itnext.io/beautify-your-golang-project-f795b4b453aa">一个例子</a></li>
<li><a href="https://hackernoon.com/golang-clean-archithecture-efd6d7c43047">Trying Clean Architecture on Golang</a>  非常好的一个示例，典范，应当时不时回过头来看看</li>
<li><a href="https://github.com/bxcodec/go-clean-arch">go-clean-arch</a>  上面帖子对应的代码</li>
<li><a href="https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html">The Clean Architecture</a> Uncle Bob</li>
</ul>


<h4>repo / 数据层如何支持数据库事务？</h4>

<p>想了好久，参考了一些别人的做法，总算有了一个相对完美的方案。关键点如下：</p>

<ul>
<li>需要抽象一个接口<code>queryUpdater</code>来代表<code>sql.DB</code>和<code>sql.TX</code></li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// queryer make sql query</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">queryer</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">NamedExec</span><span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="nx">sql</span><span class="p">.</span><span class="nx">Result</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">Get</span><span class="p">(</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">string</span><span class="p">,</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
</span><span class='line'>    <span class="nx">Select</span><span class="p">(</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">string</span><span class="p">,</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// execer execute a sql</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">execer</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">Exec</span><span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="nx">sql</span><span class="p">.</span><span class="nx">Result</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">type</span> <span class="nx">queryExecer</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">queryer</span>
</span><span class='line'>    <span class="nx">execer</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>dao对象里既有底层的sql.DB也有这个抽象的接口</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// DAO dao</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">DAO</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">db</span>     <span class="o">*</span><span class="nx">sqlx</span><span class="p">.</span><span class="nx">DB</span>
</span><span class='line'>    <span class="nx">dbConn</span> <span class="nx">queryExecer</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// NewDAO new DAO</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">NewDAO</span><span class="p">()</span> <span class="o">*</span><span class="nx">DAO</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="o">&amp;</span><span class="nx">DAO</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">db</span><span class="p">:</span>     <span class="nx">db</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">dbConn</span><span class="p">:</span> <span class="nx">db</span><span class="p">,</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>提供一个<code>WithTx</code> wrapper函数，只要是包在这个函数里的都是在一个数据库事务里执行</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// WithTx wrap txFunc in a db transaction</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">dao</span> <span class="o">*</span><span class="nx">DAO</span><span class="p">)</span> <span class="nx">WithTx</span><span class="p">(</span><span class="nx">txFunc</span> <span class="kd">func</span><span class="p">(</span><span class="nx">dao</span> <span class="o">*</span><span class="nx">DAO</span><span class="p">))</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">tx</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">Beginx</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">txDAO</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">DAO</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">dbConn</span><span class="p">:</span> <span class="nx">tx</span><span class="p">,</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">defer</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="c1">// TODO: what if err in defer?</span>
</span><span class='line'>        <span class="c1">// eg, tx.Rollback returns error</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">r</span> <span class="o">:=</span> <span class="nb">recover</span><span class="p">();</span> <span class="nx">r</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">&quot;%v&quot;</span><span class="p">,</span> <span class="nx">r</span><span class="p">)</span>
</span><span class='line'>            <span class="c1">// err := r.(error)</span>
</span><span class='line'>            <span class="nx">txErr</span> <span class="o">:=</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">Rollback</span><span class="p">()</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">txErr</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;tx rollback failed: %+v\n&quot;</span><span class="p">,</span> <span class="nx">txErr</span><span class="p">)</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;tx rollback&quot;</span><span class="p">)</span>
</span><span class='line'>            <span class="c1">// re-panic</span>
</span><span class='line'>            <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">tx</span><span class="p">.</span><span class="nx">Commit</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;tx committed&quot;</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}()</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">txFunc</span><span class="p">(</span><span class="nx">txDAO</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>事务的用法如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">jccoinDAO</span> <span class="o">:=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">NewDAO</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="nx">topup</span> <span class="o">*</span><span class="nx">model</span><span class="p">.</span><span class="nx">TopUp</span>
</span><span class='line'>
</span><span class='line'><span class="nx">jccoinDAO</span><span class="p">.</span><span class="nx">WithTx</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">dao</span> <span class="o">*</span><span class="nx">dao</span><span class="p">.</span><span class="nx">DAO</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// https://stackoverflow.com/questions/31981592/assign-struct-with-another-struct</span>
</span><span class='line'>    <span class="nx">topup</span> <span class="p">=</span> <span class="o">&amp;</span><span class="nx">model</span><span class="p">.</span><span class="nx">TopUp</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">BaseTopUp</span><span class="p">:</span>   <span class="nx">req</span><span class="p">.</span><span class="nx">BaseTopUp</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">PaidTotal</span><span class="p">:</span>   <span class="nx">req</span><span class="p">.</span><span class="nx">TopUpAmount</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">TimeCreated</span><span class="p">:</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">(),</span>
</span><span class='line'>        <span class="nx">Remark</span><span class="p">:</span>      <span class="nx">req</span><span class="p">.</span><span class="nx">Remark</span><span class="p">,</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// new topup</span>
</span><span class='line'>    <span class="c1">// idempotence support</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">isDup</span> <span class="o">:=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">NewTopUp</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">topup</span><span class="p">);</span> <span class="nx">isDup</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">topup</span> <span class="p">=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">GetTopUpByOrderNo</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">topup</span><span class="p">.</span><span class="nx">TopUpNo</span><span class="p">,</span> <span class="nx">topup</span><span class="p">.</span><span class="nx">Channel</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// get buyer_account, update balance</span>
</span><span class='line'>    <span class="nx">buyerAccount</span> <span class="o">:=</span> <span class="nx">model</span><span class="p">.</span><span class="nx">Account</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">UserID</span><span class="p">:</span>            <span class="nx">topup</span><span class="p">.</span><span class="nx">BuyerID</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">AccountType</span><span class="p">:</span>       <span class="s">&quot;BUYER&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">DeviceType</span><span class="p">:</span>        <span class="nx">topup</span><span class="p">.</span><span class="nx">DeviceType</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">SystemCode</span><span class="p">:</span>        <span class="nx">topup</span><span class="p">.</span><span class="nx">SysCode</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">IsPlatformAccount</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">AccountNumber</span><span class="p">:</span>     <span class="s">&quot;&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">Status</span><span class="p">:</span>            <span class="s">&quot;OPEN&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">Currency</span><span class="p">:</span>          <span class="s">&quot;JCC&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">TimeCreated</span><span class="p">:</span>       <span class="nx">nowFunc</span><span class="p">(),</span>
</span><span class='line'>        <span class="nx">TimeUpdated</span><span class="p">:</span>       <span class="nx">nowFunc</span><span class="p">(),</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="nx">buyerGiftAccount</span> <span class="o">:=</span> <span class="nx">buyerAccount</span>
</span><span class='line'>    <span class="nx">buyerGiftAccount</span><span class="p">.</span><span class="nx">AccountType</span> <span class="p">=</span> <span class="s">&quot;BUYERGIFT&quot;</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;buyerAccount: %+v\n&quot;</span><span class="p">,</span> <span class="nx">buyerAccount</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">ba</span> <span class="o">:=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">GetOrCreateBuyerAccount</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">buyerAccount</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="p">!</span><span class="nx">topup</span><span class="p">.</span><span class="nx">TopUpAmount</span><span class="p">.</span><span class="nx">IsZero</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">buyerBalance</span> <span class="o">:=</span> <span class="nx">ba</span><span class="p">.</span><span class="nx">Balance</span><span class="p">.</span><span class="nx">Add</span><span class="p">(</span><span class="nx">topup</span><span class="p">.</span><span class="nx">TopUpAmount</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">dao</span><span class="p">.</span><span class="nx">UpdateAccountBalance</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">ba</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span> <span class="nx">buyerBalance</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// get buyer_gift_account, update balance</span>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;giftAccount: %+v\n&quot;</span><span class="p">,</span> <span class="nx">buyerGiftAccount</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">ga</span> <span class="o">:=</span> <span class="nx">dao</span><span class="p">.</span><span class="nx">GetOrCreateBuyerAccount</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">buyerGiftAccount</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="p">!</span><span class="nx">topup</span><span class="p">.</span><span class="nx">GiftAmount</span><span class="p">.</span><span class="nx">IsZero</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">giftBalance</span> <span class="o">:=</span> <span class="nx">ga</span><span class="p">.</span><span class="nx">Balance</span><span class="p">.</span><span class="nx">Add</span><span class="p">(</span><span class="nx">topup</span><span class="p">.</span><span class="nx">GiftAmount</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">dao</span><span class="p">.</span><span class="nx">UpdateAccountBalance</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">ga</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span> <span class="nx">giftBalance</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>


<p>非事务的查询就是直接在dao上查就行：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">buyerBalance</span> <span class="o">:=</span> <span class="nx">jccoinDAO</span><span class="p">.</span><span class="nx">GetBuyerTotalBalance</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">BuyerID</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">DeviceType</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">req</span><span class="p">.</span><span class="nx">CoinsAmount</span><span class="p">.</span><span class="nx">GreaterThan</span><span class="p">(</span><span class="nx">buyerBalance</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">err</span> <span class="p">=</span> <span class="nx">ErrInsufficientBalance</span>
</span><span class='line'>    <span class="k">return</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>完美实现了代码的复用，如果不这样的话，使用事务和不使用事务时同样的查询函数要写两遍。。或者需要在参数里把<code>sql.DB</code>和<code>sql.Tx</code>都带上</p>

<p>参考：</p>

<ul>
<li><a href="https://medium.com/@jfeng45/go-microservice-with-clean-architecture-transaction-support-61eb0f886a36">Go Microservice with Clean Architecture: Transaction Support</a>  这也是个完整的方案，但感觉太复杂了。。</li>
<li><a href="https://stackoverflow.com/questions/26593867/db-transaction-in-golang">db transaction in golang</a></li>
<li><a href="https://stackoverflow.com/questions/16184238/database-sql-tx-detecting-commit-or-rollback/23502629#23502629">database/sql Tx &ndash; detecting Commit or Rollback</a></li>
<li><a href="https://github.com/agungdwiprasetyo/agungdpcms/blob/master/src/resume/usecase/usecase_impl.go#L77">agungdwiprasetyo/agungdpcms</a> 起初是这位大哥的代码给了我希望</li>
<li><a href="https://github.com/bxcodec/go-clean-arch/issues/27">Transaction #27</a> 而起初是在go-clean-arch的这个issue里看到他的回复的</li>
</ul>


<h4>cache / 缓存层？</h4>

<p>利用 <a href="https://github.com/tmrts/go-patterns/blob/master/structural/proxy.md">Proxy Pattern</a>，cache 层也实现repo层的接口，把repo包在里面，具体参考这个<a href="https://github.com/bxcodec/go-clean-arch/issues/11#issuecomment-594679205">回答</a></p>

<h3>代码Lint</h3>

<p>当前使用了<a href="https://github.com/mgechev/revive">revive</a>，号称 drop-in replacement for golint，确实名副其实，比golint强大多了~ 支持lint指定文件（夹）、排除文件夹等，支持lint参数定制化。当前支持的lint项在<a href="https://revive.run/r#redefines-builtin-id">这里</a>能看到，而且它还暴露了接口让你能自定义lint项</p>

<p>参考：</p>

<ul>
<li><a href="https://remy.io/blog/simple-tools-to-improve-your-go-code/">Simple tools to improve your Go code</a></li>
<li><a href="https://github.com/golangci/awesome-go-linters">awesome-go-linters</a></li>
<li><a href="https://github.com/mgechev/revive">revive</a>  当前使用的，非常不错</li>
</ul>


<h3>go design patterns</h3>

<p>Curated list of Go design patterns, recipes and idioms</p>

<p><a href="https://github.com/tmrts/go-patterns">go-patterns</a></p>

<h3>cmd / 临时脚本？</h3>

<p>常常有这样的需求，有一些脚本性质的任务，偶尔需要临时执行一下，这种需求在Golang项目中该怎么做呢？</p>

<p>参考了几个大型项目，比如 k8s，可以这么来做，此类任务统统放在<code>cmd</code>目录下，这个目录下可以继续建子目录，脚本就写在子目录中，但注意脚本里声明的<code>package</code>需要是<code>main</code>，如果需要执行脚本的话，可以直接执行<code>go run &lt;脚本路径&gt;</code>即可（或者把它build成一个可执行文件）</p>

<p>比如在项目中我有如下脚本：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">cmd</span>
</span><span class='line'><span class="err">└──</span> <span class="nx">kafka</span>
</span><span class='line'>    <span class="err">└──</span> <span class="nx">kafka_consume</span><span class="p">.</span><span class="k">go</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么，可以直接这样来执行它：</p>

<p><code>go run cmd/kafka/kafka_consume.go</code></p>

<blockquote><p>学到的</p></blockquote>

<p><code>package</code>的名字可以跟包含它的文件夹的名字不一样</p>

<p>参考：</p>

<ul>
<li><a href="https://github.com/golang-standards/project-layout/tree/master/cmd">golang-standards / project-layout</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/tree/master/cmd">kubernetes/cmd</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[error code vs exception]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/01/26/error-code-vs-exception/"/>
    <updated>2021-01-26T18:14:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/01/26/error-code-vs-exception</id>
    <content type="html"><![CDATA[<p>error-code OR exception, it&rsquo;s a question.</p>

<p>然而答案是，you should use both，难点在于确定在什么场景下用什么</p>

<h3>适合用 exception 的情况</h3>

<ul>
<li>发生的错误无法处理时</li>
</ul>


<p>In some situations <strong>you can&rsquo;t do anything with the error code</strong>, and you just need to handle it in an upper level in the call stack, usually just log the error, display something to the user or close the program. In these cases, error codes would require you to bubble up the error codes manually level by level which is obviously much easier to do with exceptions. The point is that this is for unexpected and unhandleable situations. 与其将错误一层一层的 bubble up，不如将它一次性抛到最顶层</p>

<ul>
<li>In some situations you have a sequence of commands, and if any of them fail you should cleanup/dispose resources. 当有一堆调用要执行，而且每一个调用都有可能报错时，很明显用 exception 更方便，而不是每次调用后都要 handle 一次可能的error。</li>
<li>In constructors you can only throw exceptions. 在初始化时若出错一律直接抛 exception，遵循速错原则</li>
</ul>


<h3>适合用 error-code 的情况</h3>

<p>一般来说，除了上面适合使用 exception 的情况，其它任何时候都应该使用 error-code。</p>

<p>in all other situations where you&rsquo;re returning some information on which the caller CAN/SHOULD take some action, using return codes is probably a better alternative. This includes all expected &ldquo;errors&rdquo;, because probably they should be handled by the immediate caller, and will hardly need to be bubbled up too many levels up in the stack.</p>

<h3>sum up</h3>

<p>To sum it up:</p>

<ul>
<li><p>I like to use exceptions when I have an unexpected situation, in which there&rsquo;s not much to do, and usually we want to abort a large block of code or even the whole operation or program. This is like the old &ldquo;on error goto&rdquo;.</p></li>
<li><p>I like to use return codes when I have expected situations in which the caller code can/should take some action. This includes most business methods, APIs, validations, and so on.</p></li>
</ul>


<h3>Unchecked Exceptions 的问题</h3>

<p>首先，啥是 unchecked exception？</p>

<p>A quick recap. In an unchecked exceptions model, <strong>you throw and catch exceptions, without it being part of the type system or a function’s signature</strong>.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="c1">// Foo throws an unhandled exception:</span>
</span><span class='line'><span class="kt">void</span> <span class="nf">Foo</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">throw</span> <span class="k">new</span> <span class="nf">Exception</span><span class="o">(...);</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Bar calls Foo, and handles that exception:</span>
</span><span class='line'><span class="kt">void</span> <span class="nf">Bar</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">try</span> <span class="o">{</span>
</span><span class='line'>        <span class="n">Foo</span><span class="o">();</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>    <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>        <span class="c1">// Handle the error.</span>
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Baz also calls Foo, but does not handle that exception:</span>
</span><span class='line'><span class="kt">void</span> <span class="nf">Baz</span><span class="o">()</span> <span class="o">{</span>
</span><span class='line'>    <span class="n">Foo</span><span class="o">();</span> <span class="c1">// Let the error escape to our callers.</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>问题在哪？</p>

<p>In this model, <strong>any function call – and sometimes any statement – can throw an exception, transferring control non-locally somewhere else. Where? Who knows.</strong> There are no annotations or type system artifacts to guide your analysis. As a result, it’s difficult for anyone to reason about a program’s state at the time of the throw, the state changes that occur while that exception is propagated up the call stack – and possibly across threads in a concurrent program – and the resulting state by the time it gets caught or goes unhandled. 任何地方都有可能抛出exception，使得对于代码的分析变得非常困难。（我理解，这是因为没有限制，所以大家可以随心所欲的写，因而也非常难于理解。若大家在写的时候都遵循一定的规则，理论上也是可以写出易懂的代码，但每个人的水平不一样，对待代码的态度也不一样，在工程上没有限制的情况下，必然会造成“百花齐放”的现象）</p>

<p>非常经典的分析：为什么 exception 在业界如此流行？</p>

<blockquote><p>It’s of course possible to try. Doing so requires reading API documentation, doing manual audits of the code, leaning heavily on code reviews, and a healthy dose of luck. The language isn’t helping you out one bit here. Because failures are rare, this tends not to be as utterly disastrous as it sounds. My conclusion is that’s why many people in the industry think unchecked exceptions are “good enough.” They stay out of your way for the common success paths and, because most people don’t write robust error handling code in non-systems programs, throwing an exception usually gets you out of a pickle fast. Catching and then proceeding often works too. No harm, no foul. Statistically speaking, programs “work.”</p></blockquote>

<p>exception 的通病（不管是checked还是unchecked）</p>

<blockquote><p>Another thing most exception systems get wrong is encouraging too coarse a granularity of handling errors. Proponents of return codes love that error handling is localized to a specific function call. (I do too!) In exception handling systems, it’s all too easy to slap a coarse-grained try/catch block around some huge hunk of code, without carefully reacting to individual failures. That produces brittle code that’s almost certainly wrong; if not today, then after the inevitable refactoring that will occur down the road. A lot of this has to do with having the right syntaxes. 一般人通常做法是一个<code>try ... catch</code> wrap 住一大段代码，简单是简单，但一般会埋坑</p></blockquote>

<p>参考：</p>

<ul>
<li><a href="https://stackoverflow.com/questions/253314/conventions-for-exceptions-or-error-codes">Conventions for exceptions or error codes</a></li>
<li><a href="http://joeduffyblog.com/2016/02/07/the-error-model/">The Error Model</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[sarama源码分析Part I]]></title>
    <link href="http://cs50Mu.github.io/blog/2021/01/22/source-code-of-sarama-part-i/"/>
    <updated>2021-01-22T20:07:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2021/01/22/source-code-of-sarama-part-i</id>
    <content type="html"><![CDATA[<p><a href="https://github.com/Shopify/sarama">Shopify / sarama</a></p>

<p>以下代码分析基于此快照的代码：2a5254c26ef606cd9a587</p>

<h3>生产者如何发送消息</h3>

<p>这一篇帖子已经分析的非常好了：<a href="https://juejin.cn/post/6866316565348876296#heading-6">Sarama生产者是如何工作的</a></p>

<p>在evernote的备份：<a href="https://www.evernote.com/shard/s112/nl/12260165/5ab3f175-2851-4069-b3b4-b6dc20f7f203/">Sarama生产者是如何工作的</a></p>

<p>也可以看下官方wiki里关于producer实现的说明：<a href="https://github.com/Shopify/sarama/wiki/Producer-implementation">Producer implementation</a></p>

<h4>从提问题中来学习</h4>

<blockquote><p>kafka client 是如何与 broker server 交互的？</p></blockquote>

<p>以 获取 metadata 为例（一个client启动后先要做的就是获取metadata，比如，有哪些topic，某个topic有几个partition，哪个broker是leader等）：</p>

<p>可以看到内部调用的是<code>sendAndReceive</code>这个方法</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">//GetMetadata send a metadata request and returns a metadata response or error</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="nx">GetMetadata</span><span class="p">(</span><span class="nx">request</span> <span class="o">*</span><span class="nx">MetadataRequest</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">MetadataResponse</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">response</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="nx">MetadataResponse</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">err</span> <span class="o">:=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">sendAndReceive</span><span class="p">(</span><span class="nx">request</span><span class="p">,</span> <span class="nx">response</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">return</span> <span class="nx">response</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>




<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="nx">sendAndReceive</span><span class="p">(</span><span class="nx">req</span> <span class="nx">protocolBody</span><span class="p">,</span> <span class="nx">res</span> <span class="nx">versionedDecoder</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// send 返回一个 promise</span>
</span><span class='line'>      <span class="nx">promise</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">send</span><span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span> <span class="o">!=</span> <span class="kc">nil</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 当配置`RequiredAcks`=0(即不关注server返回)时，返回的promise会是nil</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">promise</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 然后就等待这个 promise</span>
</span><span class='line'>      <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">buf</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">promise</span><span class="p">.</span><span class="nx">packets</span><span class="p">:</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">versionedDecode</span><span class="p">(</span><span class="nx">buf</span><span class="p">,</span> <span class="nx">res</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">version</span><span class="p">())</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">err</span> <span class="p">=</span> <span class="o">&lt;-</span><span class="nx">promise</span><span class="p">.</span><span class="nx">errors</span><span class="p">:</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>既然是返回了promise，说明<code>send</code>方法是异步的：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 为节省篇幅，省略了一些错误处理的展示</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="nx">send</span><span class="p">(</span><span class="nx">rb</span> <span class="nx">protocolBody</span><span class="p">,</span> <span class="nx">promiseResponse</span> <span class="kt">bool</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">responsePromise</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">defer</span> <span class="nx">b</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="o">...</span><span class="p">.</span>
</span><span class='line'>      <span class="c1">// request的correlationID</span>
</span><span class='line'>      <span class="nx">req</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">request</span><span class="p">{</span><span class="nx">correlationID</span><span class="p">:</span> <span class="nx">b</span><span class="p">.</span><span class="nx">correlationID</span><span class="p">,</span> <span class="nx">clientID</span><span class="p">:</span> <span class="nx">b</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">ClientID</span><span class="p">,</span> <span class="nx">body</span><span class="p">:</span> <span class="nx">rb</span><span class="p">}</span>
</span><span class='line'>      <span class="nx">buf</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">encode</span><span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">b</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">MetricRegistry</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">requestTime</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">()</span>
</span><span class='line'>      <span class="c1">// Will be decremented in responseReceiver (except error or request with NoResponse)</span>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">addRequestInFlightMetrics</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 关键在这里了，这里把请求发出去了</span>
</span><span class='line'>      <span class="c1">// 最终会调用`conn.Write`</span>
</span><span class='line'>      <span class="nx">bytes</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="nx">buf</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">updateOutgoingCommunicationMetrics</span><span class="p">(</span><span class="nx">bytes</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">addRequestInFlightMetrics</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">correlationID</span><span class="o">++</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">promiseResponse</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Record request latency without the response</span>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">updateRequestLatencyAndInFlightMetrics</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Since</span><span class="p">(</span><span class="nx">requestTime</span><span class="p">))</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 组装了一个promise，然后把它发给了一个channel：b.responses</span>
</span><span class='line'>      <span class="c1">// 注意这个promise里也有两个channel，它是用来返回结果的</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 关联上了req.correlationID</span>
</span><span class='line'>      <span class="c1">// 通过这个correlationID来关联req和response</span>
</span><span class='line'>      <span class="nx">promise</span> <span class="o">:=</span> <span class="nx">responsePromise</span><span class="p">{</span><span class="nx">requestTime</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">correlationID</span><span class="p">,</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">),</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">error</span><span class="p">)}</span>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">responses</span> <span class="o">&lt;-</span> <span class="nx">promise</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="o">&amp;</span><span class="nx">promise</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么，很明显的，一定有人在监听这个<code>b.responses</code>了，是在<code>responseReceiver</code>方法中：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="nx">responseReceiver</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">dead</span> <span class="kt">error</span>
</span><span class='line'>      <span class="nx">header</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 注意是一个循环读取消息，即使没有消息</span>
</span><span class='line'>      <span class="c1">// 也会等待在这里，除非有人关闭了这个channel</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">response</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">b</span><span class="p">.</span><span class="nx">responses</span> <span class="p">{</span>
</span><span class='line'>          <span class="o">...</span>
</span><span class='line'>          <span class="c1">// 读header</span>
</span><span class='line'>          <span class="nx">bytesReadHeader</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">readFull</span><span class="p">(</span><span class="nx">header</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">requestLatency</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Since</span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">requestTime</span><span class="p">)</span>
</span><span class='line'>          <span class="o">...</span><span class="p">.</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">decodedHeader</span> <span class="o">:=</span> <span class="nx">responseHeader</span><span class="p">{}</span>
</span><span class='line'>          <span class="nx">err</span> <span class="p">=</span> <span class="nx">decode</span><span class="p">(</span><span class="nx">header</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">decodedHeader</span><span class="p">)</span>
</span><span class='line'>          <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>          <span class="c1">// 这里会校验收到的消息是否是所预期的，若对不上直接抛弃，继续处理下一条</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">decodedHeader</span><span class="p">.</span><span class="nx">correlationID</span> <span class="o">!=</span> <span class="nx">response</span><span class="p">.</span><span class="nx">correlationID</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">b</span><span class="p">.</span><span class="nx">updateIncomingCommunicationMetrics</span><span class="p">(</span><span class="nx">bytesReadHeader</span><span class="p">,</span> <span class="nx">requestLatency</span><span class="p">)</span>
</span><span class='line'>              <span class="c1">// TODO if decoded ID &lt; cur ID, discard until we catch up</span>
</span><span class='line'>              <span class="c1">// TODO if decoded ID &gt; cur ID, save it so when cur ID catches up we have a response</span>
</span><span class='line'>              <span class="nx">dead</span> <span class="p">=</span> <span class="nx">PacketDecodingError</span><span class="p">{</span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">&quot;correlation ID didn&#39;t match, wanted %d, got %d&quot;</span><span class="p">,</span> <span class="nx">response</span><span class="p">.</span><span class="nx">correlationID</span><span class="p">,</span> <span class="nx">decodedHeader</span><span class="p">.</span><span class="nx">correlationID</span><span class="p">)}</span>
</span><span class='line'>              <span class="nx">response</span><span class="p">.</span><span class="nx">errors</span> <span class="o">&lt;-</span> <span class="nx">dead</span>
</span><span class='line'>              <span class="k">continue</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="c1">// 读body</span>
</span><span class='line'>          <span class="nx">buf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="nx">decodedHeader</span><span class="p">.</span><span class="nx">length</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">bytesReadBody</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">readFull</span><span class="p">(</span><span class="nx">buf</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">updateIncomingCommunicationMetrics</span><span class="p">(</span><span class="nx">bytesReadHeader</span><span class="o">+</span><span class="nx">bytesReadBody</span><span class="p">,</span> <span class="nx">requestLatency</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">dead</span> <span class="p">=</span> <span class="nx">err</span>
</span><span class='line'>              <span class="nx">response</span><span class="p">.</span><span class="nx">errors</span> <span class="o">&lt;-</span> <span class="nx">err</span>
</span><span class='line'>              <span class="k">continue</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="c1">// 将读到的body字节通过 channel 返回</span>
</span><span class='line'>          <span class="nx">response</span><span class="p">.</span><span class="nx">packets</span> <span class="o">&lt;-</span> <span class="nx">buf</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nb">close</span><span class="p">(</span><span class="nx">b</span><span class="p">.</span><span class="nx">done</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么，这个<code>responseReceiver</code>方法是什么时候调用的呢？是在open 一个 broker 的时候：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">b</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="nx">Open</span><span class="p">(</span><span class="nx">conf</span> <span class="o">*</span><span class="nx">Config</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">atomic</span><span class="p">.</span><span class="nx">CompareAndSwapInt32</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">b</span><span class="p">.</span><span class="nx">opened</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">ErrAlreadyConnected</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">conf</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">conf</span> <span class="p">=</span> <span class="nx">NewConfig</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">err</span> <span class="o">:=</span> <span class="nx">conf</span><span class="p">.</span><span class="nx">Validate</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">b</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">go</span> <span class="nx">withRecover</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">defer</span> <span class="nx">b</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">dialer</span> <span class="o">:=</span> <span class="nx">net</span><span class="p">.</span><span class="nx">Dialer</span><span class="p">{</span>
</span><span class='line'>              <span class="nx">Timeout</span><span class="p">:</span>   <span class="nx">conf</span><span class="p">.</span><span class="nx">Net</span><span class="p">.</span><span class="nx">DialTimeout</span><span class="p">,</span>
</span><span class='line'>              <span class="nx">KeepAlive</span><span class="p">:</span> <span class="nx">conf</span><span class="p">.</span><span class="nx">Net</span><span class="p">.</span><span class="nx">KeepAlive</span><span class="p">,</span>
</span><span class='line'>              <span class="nx">LocalAddr</span><span class="p">:</span> <span class="nx">conf</span><span class="p">.</span><span class="nx">Net</span><span class="p">.</span><span class="nx">LocalAddr</span><span class="p">,</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">conn</span> <span class="p">=</span> <span class="nx">newBufConn</span><span class="p">(</span><span class="nx">b</span><span class="p">.</span><span class="nx">conn</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">conf</span> <span class="p">=</span> <span class="nx">conf</span>
</span><span class='line'>
</span><span class='line'>          <span class="o">...</span><span class="p">.</span>
</span><span class='line'>
</span><span class='line'>          <span class="k">if</span> <span class="nx">b</span><span class="p">.</span><span class="nx">id</span> <span class="o">&gt;=</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">b</span><span class="p">.</span><span class="nx">registerMetrics</span><span class="p">()</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="o">...</span><span class="p">.</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">done</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">bool</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">b</span><span class="p">.</span><span class="nx">responses</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="nx">responsePromise</span><span class="p">,</span> <span class="nx">b</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Net</span><span class="p">.</span><span class="nx">MaxOpenRequests</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>          <span class="k">if</span> <span class="nx">b</span><span class="p">.</span><span class="nx">id</span> <span class="o">&gt;=</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;Connected to broker at %s (registered as #%d)\n&quot;</span><span class="p">,</span> <span class="nx">b</span><span class="p">.</span><span class="nx">addr</span><span class="p">,</span> <span class="nx">b</span><span class="p">.</span><span class="nx">id</span><span class="p">)</span>
</span><span class='line'>          <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;Connected to broker at %s (unregistered)\n&quot;</span><span class="p">,</span> <span class="nx">b</span><span class="p">.</span><span class="nx">addr</span><span class="p">)</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// 这里单独起了一个协程来处理</span>
</span><span class='line'>          <span class="k">go</span> <span class="nx">withRecover</span><span class="p">(</span><span class="nx">b</span><span class="p">.</span><span class="nx">responseReceiver</span><span class="p">)</span>
</span><span class='line'>      <span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>


<p>什么时候会open broker呢？在新建Client的时候</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">NewClient</span><span class="p">(</span><span class="nx">addrs</span> <span class="p">[]</span><span class="kt">string</span><span class="p">,</span> <span class="nx">conf</span> <span class="o">*</span><span class="nx">Config</span><span class="p">)</span> <span class="p">(</span><span class="nx">Client</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;Initializing new client&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="o">...</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">conf</span><span class="p">.</span><span class="nx">Metadata</span><span class="p">.</span><span class="nx">Full</span> <span class="p">{</span>
</span><span class='line'>        <span class="c1">// do an initial fetch of all cluster metadata by specifying an empty list of topics</span>
</span><span class='line'>        <span class="c1">// 这里会从broker拉取一次metadata</span>
</span><span class='line'>        <span class="c1">// 在此过程中首先会先open一个broker</span>
</span><span class='line'>        <span class="nx">err</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">RefreshMetadata</span><span class="p">()</span>
</span><span class='line'>        <span class="k">switch</span> <span class="nx">err</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="kc">nil</span><span class="p">:</span>
</span><span class='line'>            <span class="k">break</span>
</span><span class='line'>        <span class="k">case</span> <span class="nx">ErrLeaderNotAvailable</span><span class="p">,</span> <span class="nx">ErrReplicaNotAvailable</span><span class="p">,</span> <span class="nx">ErrTopicAuthorizationFailed</span><span class="p">,</span> <span class="nx">ErrClusterAuthorizationFailed</span><span class="p">:</span>
</span><span class='line'>            <span class="c1">// indicates that maybe part of the cluster is down, but is not fatal to creating the client</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>        <span class="k">default</span><span class="p">:</span>
</span><span class='line'>            <span class="nb">close</span><span class="p">(</span><span class="nx">client</span><span class="p">.</span><span class="nx">closed</span><span class="p">)</span> <span class="c1">// we haven&#39;t started the background updater yet, so we have to do this manually</span>
</span><span class='line'>            <span class="nx">_</span> <span class="p">=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>            <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="c1">// 这里启动后台定时拉取metadata的任务</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">withRecover</span><span class="p">(</span><span class="nx">client</span><span class="p">.</span><span class="nx">backgroundMetadataUpdater</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;Successfully initialized new client&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nx">client</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>    <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>总结一下，模式是：制造任务，扔进队列（channel），新启动一个协程来监听队列（channel），有新任务就做，任务的完成情况是通过任务结构体中的channel来跟调用方完成通信的</p>

<blockquote><p>通过 kafka client 生产(produce)一条消息，需要经过哪些步骤？</p></blockquote>

<p>sarama 有两种 producer: syncProducer 和 asyncProducer，syncProducer 只是 asyncProducer 的一个简单封装，下面只介绍 asyncProducer 的发送消息的流程：</p>

<p>一条消息在真正被发到网络上之前，会流经以下结构：</p>

<p>asyncProducer(singleton) &mdash;> topicProducer(one per topic) &mdash;> partitionProducer(one per partition) &mdash;> brokerProducer(one per broker)</p>

<p>消息的传递是通过 channel，每个结构体在处理完消息后，会将它丢进下一个结构体在监听的channel中。</p>

<p>重点关注下<code>BrokerProducer</code>，这里起了两个goroutine，<code>bp.run</code>负责将消息打包（压缩）成一个set，另一个goroutine负责将打包好的set真正发出去</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// one per broker; also constructs an associated flusher</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">asyncProducer</span><span class="p">)</span> <span class="nx">newBrokerProducer</span><span class="p">(</span><span class="nx">broker</span> <span class="o">*</span><span class="nx">Broker</span><span class="p">)</span> <span class="o">*</span><span class="nx">brokerProducer</span> <span class="p">{</span>
</span><span class='line'>    <span class="kd">var</span> <span class="p">(</span>
</span><span class='line'>        <span class="nx">input</span>     <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="o">*</span><span class="nx">ProducerMessage</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">bridge</span>    <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="o">*</span><span class="nx">produceSet</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">responses</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="o">*</span><span class="nx">brokerProducerResponse</span><span class="p">)</span>
</span><span class='line'>    <span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">bp</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">brokerProducer</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">parent</span><span class="p">:</span>         <span class="nx">p</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">broker</span><span class="p">:</span>         <span class="nx">broker</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">input</span><span class="p">:</span>          <span class="nx">input</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">output</span><span class="p">:</span>         <span class="nx">bridge</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">responses</span><span class="p">:</span>      <span class="nx">responses</span><span class="p">,</span>
</span><span class='line'>        <span class="nx">stopchan</span><span class="p">:</span>       <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{}),</span>
</span><span class='line'>        <span class="nx">buffer</span><span class="p">:</span>         <span class="nx">newProduceSet</span><span class="p">(</span><span class="nx">p</span><span class="p">),</span>
</span><span class='line'>        <span class="nx">currentRetries</span><span class="p">:</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kd">map</span><span class="p">[</span><span class="kt">int32</span><span class="p">]</span><span class="kt">error</span><span class="p">),</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">withRecover</span><span class="p">(</span><span class="nx">bp</span><span class="p">.</span><span class="nx">run</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// minimal bridge to make the network response `select`able</span>
</span><span class='line'>    <span class="c1">// 注意这里是按set来发的，消息走到这里时已经被整合成一个batch了</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">withRecover</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">for</span> <span class="nx">set</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">bridge</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">request</span> <span class="o">:=</span> <span class="nx">set</span><span class="p">.</span><span class="nx">buildRequest</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">response</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">Produce</span><span class="p">(</span><span class="nx">request</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">responses</span> <span class="o">&lt;-</span> <span class="o">&amp;</span><span class="nx">brokerProducerResponse</span><span class="p">{</span>
</span><span class='line'>                <span class="nx">set</span><span class="p">:</span> <span class="nx">set</span><span class="p">,</span>
</span><span class='line'>                <span class="nx">err</span><span class="p">:</span> <span class="nx">err</span><span class="p">,</span>
</span><span class='line'>                <span class="nx">res</span><span class="p">:</span> <span class="nx">response</span><span class="p">,</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>        <span class="nb">close</span><span class="p">(</span><span class="nx">responses</span><span class="p">)</span>
</span><span class='line'>    <span class="p">})</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="nx">p</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Producer</span><span class="p">.</span><span class="nx">Retry</span><span class="p">.</span><span class="nx">Max</span> <span class="o">&lt;=</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">bp</span><span class="p">.</span><span class="nx">abandoned</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nx">bp</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>参考：<a href="https://github.com/Shopify/sarama/wiki/Producer-implementation#message-flow">Message Flow</a></p>

<blockquote><p>如何保持发送的消息有序？</p></blockquote>

<p>基本思想是：需要重发的消息会被重新扔到队列中，不过producer对于重发的消息和正常的消息的处理策略不一样：</p>

<ul>
<li>从发现有重发的消息起，优先发送<code>retriedMsg</code></li>
<li>正常的消息会被缓存起来</li>
<li>当确认所有的<code>retriedMsg</code>都发送成功后，再将这期间缓存的正常消息一次性发送出去</li>
</ul>


<p>代码其实比刚刚的描述复杂多了，它还有一个“水位线(Watermark)”的概念，允许重试多次，对于重试同样次数的消息被放在同一个level的buffer中</p>

<p>Maintaining Order</p>

<p>Maintaining the order of messages when a retry occurs is an additional challenge. When a brokerProducer triggers a retry, the following events occur, strictly in this order:</p>

<ul>
<li><p>the messages to retry are sent to the retryHandler</p></li>
<li><p>the brokerProducer sets a flag for the given topic/partition; while this flag is set any further such messages (which may have already been in the pipeline) will be immediately sent to the retryHandler</p></li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 此处逻辑位于 func (bp *brokerProducer) handleSuccess 方法中</span>
</span><span class='line'>        <span class="nx">sent</span><span class="p">.</span><span class="nx">eachPartition</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">topic</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">partition</span> <span class="kt">int32</span><span class="p">,</span> <span class="nx">pSet</span> <span class="o">*</span><span class="nx">partitionSet</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">block</span> <span class="o">:=</span> <span class="nx">response</span><span class="p">.</span><span class="nx">GetBlock</span><span class="p">(</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">partition</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">block</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="c1">// handled in the previous &quot;eachPartition&quot; loop</span>
</span><span class='line'>                <span class="k">return</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="k">switch</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span> <span class="p">{</span>
</span><span class='line'>            <span class="k">case</span> <span class="nx">ErrInvalidMessage</span><span class="p">,</span> <span class="nx">ErrUnknownTopicOrPartition</span><span class="p">,</span> <span class="nx">ErrLeaderNotAvailable</span><span class="p">,</span> <span class="nx">ErrNotLeaderForPartition</span><span class="p">,</span>
</span><span class='line'>                <span class="nx">ErrRequestTimedOut</span><span class="p">,</span> <span class="nx">ErrNotEnoughReplicas</span><span class="p">,</span> <span class="nx">ErrNotEnoughReplicasAfterAppend</span><span class="p">:</span>
</span><span class='line'>                <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/broker/%d state change to [retrying] on %s/%d because %v\n&quot;</span><span class="p">,</span>
</span><span class='line'>                    <span class="nx">bp</span><span class="p">.</span><span class="nx">broker</span><span class="p">.</span><span class="nx">ID</span><span class="p">(),</span> <span class="nx">topic</span><span class="p">,</span> <span class="nx">partition</span><span class="p">,</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span><span class="p">)</span>
</span><span class='line'>                <span class="k">if</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">topic</span><span class="p">]</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                    <span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">topic</span><span class="p">]</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">int32</span><span class="p">]</span><span class="kt">error</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span>
</span><span class='line'>                <span class="c1">// 这里是所谓的`sets a flag for the given topic/partition`</span>
</span><span class='line'>                <span class="c1">// 用于标识这个topic/partition是有问题的</span>
</span><span class='line'>                <span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">topic</span><span class="p">][</span><span class="nx">partition</span><span class="p">]</span> <span class="p">=</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span>
</span><span class='line'>                <span class="c1">// 以下为将消息发送至 retryHandler</span>
</span><span class='line'>                <span class="k">if</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Producer</span><span class="p">.</span><span class="nx">Idempotent</span> <span class="p">{</span>
</span><span class='line'>                    <span class="k">go</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">retryBatch</span><span class="p">(</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">partition</span><span class="p">,</span> <span class="nx">pSet</span><span class="p">,</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'>                    <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">retryMessages</span><span class="p">(</span><span class="nx">pSet</span><span class="p">.</span><span class="nx">msgs</span><span class="p">,</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span>
</span><span class='line'>                <span class="c1">// dropping the following messages has the side effect of incrementing their retry count</span>
</span><span class='line'>                <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">retryMessages</span><span class="p">(</span><span class="nx">bp</span><span class="p">.</span><span class="nx">buffer</span><span class="p">.</span><span class="nx">dropPartition</span><span class="p">(</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">partition</span><span class="p">),</span> <span class="nx">block</span><span class="p">.</span><span class="nx">Err</span><span class="p">)</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>        <span class="p">})</span>
</span></code></pre></td></tr></table></div></figure>




<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 此处逻辑位于 func (bp *brokerProducer) run 方法</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">reason</span> <span class="o">:=</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">needsRetry</span><span class="p">(</span><span class="nx">msg</span><span class="p">);</span> <span class="nx">reason</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">retryMessage</span><span class="p">(</span><span class="nx">msg</span><span class="p">,</span> <span class="nx">reason</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>                <span class="k">if</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">closing</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">flags</span><span class="o">&amp;</span><span class="nx">fin</span> <span class="o">==</span> <span class="nx">fin</span> <span class="p">{</span>
</span><span class='line'>                    <span class="c1">// we were retrying this partition but we can start processing again</span>
</span><span class='line'>                    <span class="nb">delete</span><span class="p">(</span><span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">],</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">)</span>
</span><span class='line'>                    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/broker/%d state change to [closed] on %s/%d\n&quot;</span><span class="p">,</span>
</span><span class='line'>                        <span class="nx">bp</span><span class="p">.</span><span class="nx">broker</span><span class="p">.</span><span class="nx">ID</span><span class="p">(),</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">,</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>                <span class="k">continue</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">bp</span> <span class="o">*</span><span class="nx">brokerProducer</span><span class="p">)</span> <span class="nx">needsRetry</span><span class="p">(</span><span class="nx">msg</span> <span class="o">*</span><span class="nx">ProducerMessage</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">closing</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">closing</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">][</span><span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">]</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li><p>eventually the first retried message reaches its partitionProducer</p></li>
<li><p>the partitionProducer sends off a special &ldquo;chaser&rdquo; message and releases its reference to the old broker</p></li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 此处逻辑位于 func (pp *partitionProducer) dispatch 中</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span> <span class="p">&gt;</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="p">{</span>
</span><span class='line'>            <span class="c1">// a new, higher, retry level; handle it and then back off</span>
</span><span class='line'>            <span class="nx">pp</span><span class="p">.</span><span class="nx">newHighWatermark</span><span class="p">(</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">)</span>
</span><span class='line'>            <span class="nx">pp</span><span class="p">.</span><span class="nx">backoff</span><span class="p">(</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">pp</span> <span class="o">*</span><span class="nx">partitionProducer</span><span class="p">)</span> <span class="nx">newHighWatermark</span><span class="p">(</span><span class="nx">hwm</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d state change to [retrying-%d]\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">hwm</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="p">=</span> <span class="nx">hwm</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// send off a fin so that we know when everything &quot;in between&quot; has made it</span>
</span><span class='line'>    <span class="c1">// back to us and we can safely flush the backlog (otherwise we risk re-ordering messages)</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">expectChaser</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">inFlight</span><span class="p">.</span><span class="nx">Add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1">// we&#39;re generating a fin message; track it so we don&#39;t shut down while it&#39;s still inflight</span>
</span><span class='line'>    <span class="c1">// 注意此处的retries做了减一操作</span>
</span><span class='line'>    <span class="c1">// 这样的话，当brokerProducer收到这个fin消息，并重新retry到partitionProducer中时</span>
</span><span class='line'>    <span class="c1">// 才会满足msg.retries == highWatermark 从而被识别</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span><span class="p">.</span><span class="nx">input</span> <span class="o">&lt;-</span> <span class="o">&amp;</span><span class="nx">ProducerMessage</span><span class="p">{</span><span class="nx">Topic</span><span class="p">:</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">Partition</span><span class="p">:</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">flags</span><span class="p">:</span> <span class="nx">fin</span><span class="p">,</span> <span class="nx">retries</span><span class="p">:</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="o">-</span> <span class="mi">1</span><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// a new HWM means that our current broker selection is out of date</span>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d abandoning broker %d\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">leader</span><span class="p">.</span><span class="nx">ID</span><span class="p">())</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">unrefBrokerProducer</span><span class="p">(</span><span class="nx">pp</span><span class="p">.</span><span class="nx">leader</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>the partitionProducer updates its metadata, opens a connection to the new broker, and sends the retried message down the new path</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 依然在 func (pp *partitionProducer) dispatch 中</span>
</span><span class='line'>        <span class="c1">// if we made it this far then the current msg contains real data, and can be sent to the next goroutine</span>
</span><span class='line'>        <span class="c1">// without breaking any of our ordering guarantees</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">updateLeader</span><span class="p">();</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">returnError</span><span class="p">(</span><span class="nx">msg</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">backoff</span><span class="p">(</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">)</span>
</span><span class='line'>                <span class="k">continue</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d selected broker %d\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">leader</span><span class="p">.</span><span class="nx">ID</span><span class="p">())</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// Now that we know we have a broker to actually try and send this message to, generate the sequence</span>
</span><span class='line'>        <span class="c1">// number for it.</span>
</span><span class='line'>        <span class="c1">// All messages being retried (sent or not) have already had their retry count updated</span>
</span><span class='line'>        <span class="c1">// Also, ignore &quot;special&quot; syn/fin messages used to sync the brokerProducer and the topicProducer.</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Producer</span><span class="p">.</span><span class="nx">Idempotent</span> <span class="o">&amp;&amp;</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">flags</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">msg</span><span class="p">.</span><span class="nx">sequenceNumber</span><span class="p">,</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">producerEpoch</span> <span class="p">=</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">txnmgr</span><span class="p">.</span><span class="nx">getAndIncrementSequenceNumber</span><span class="p">(</span><span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">,</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">)</span>
</span><span class='line'>            <span class="nx">msg</span><span class="p">.</span><span class="nx">hasSequence</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span><span class="p">.</span><span class="nx">input</span> <span class="o">&lt;-</span> <span class="nx">msg</span>
</span><span class='line'>    <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>the partitionProducer continues handling incoming messages &ndash; retried messages get sent to the new broker, while new messages are held in a queue to preserver ordering</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 依然在 func (pp *partitionProducer) dispatch 中</span>
</span><span class='line'>
</span><span class='line'>        <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>            <span class="c1">// we are retrying something (else highWatermark would be 0) but this message is not a *new* retry level</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span> <span class="p">&lt;</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="p">{</span>
</span><span class='line'>                <span class="c1">// in fact this message is not even the current retry level, so buffer it for now (unless it&#39;s a just a fin)</span>
</span><span class='line'>                <span class="k">if</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">flags</span><span class="o">&amp;</span><span class="nx">fin</span> <span class="o">==</span> <span class="nx">fin</span> <span class="p">{</span>
</span><span class='line'>                    <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">].</span><span class="nx">expectChaser</span> <span class="p">=</span> <span class="kc">false</span>
</span><span class='line'>                    <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">inFlight</span><span class="p">.</span><span class="nx">Done</span><span class="p">()</span> <span class="c1">// this fin is now handled and will be garbage collected</span>
</span><span class='line'>                <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'>                <span class="c1">// 新消息暂存在buffer中</span>
</span><span class='line'>                    <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">].</span><span class="nx">buf</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">retries</span><span class="p">].</span><span class="nx">buf</span><span class="p">,</span> <span class="nx">msg</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span>
</span><span class='line'>                <span class="k">continue</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// 重试的消息会发往新开启的broker中</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li>the brokerProducer sees the chaser message; it clears the flag it originally sent, and &ldquo;retries&rdquo; the chaser message</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 此处逻辑位于 func (bp *brokerProducer) run 方法</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">reason</span> <span class="o">:=</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">needsRetry</span><span class="p">(</span><span class="nx">msg</span><span class="p">);</span> <span class="nx">reason</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="c1">// ”重发”</span>
</span><span class='line'>                <span class="nx">bp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">retryMessage</span><span class="p">(</span><span class="nx">msg</span><span class="p">,</span> <span class="nx">reason</span><span class="p">)</span>
</span><span class='line'>                <span class="c1">// clear flag</span>
</span><span class='line'>                <span class="k">if</span> <span class="nx">bp</span><span class="p">.</span><span class="nx">closing</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">flags</span><span class="o">&amp;</span><span class="nx">fin</span> <span class="o">==</span> <span class="nx">fin</span> <span class="p">{</span>
</span><span class='line'>                    <span class="c1">// we were retrying this partition but we can start processing again</span>
</span><span class='line'>                    <span class="nb">delete</span><span class="p">(</span><span class="nx">bp</span><span class="p">.</span><span class="nx">currentRetries</span><span class="p">[</span><span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">],</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">)</span>
</span><span class='line'>                    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/broker/%d state change to [closed] on %s/%d\n&quot;</span><span class="p">,</span>
</span><span class='line'>                        <span class="nx">bp</span><span class="p">.</span><span class="nx">broker</span><span class="p">.</span><span class="nx">ID</span><span class="p">(),</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Topic</span><span class="p">,</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">Partition</span><span class="p">)</span>
</span><span class='line'>                <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>                <span class="k">continue</span>
</span><span class='line'>            <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<ul>
<li><p>the partitionProducer sees the retried chaser message (indicating that it has seen the last retried message)  这是最后一条 retried msg</p></li>
<li><p>the partitionProducer flushes the backlog of &ldquo;new&rdquo; messages to the new broker and resumes normal processing</p></li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 此处逻辑在 func (pp *partitionProducer) dispatch 中</span>
</span><span class='line'>            <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="nx">msg</span><span class="p">.</span><span class="nx">flags</span><span class="o">&amp;</span><span class="nx">fin</span> <span class="o">==</span> <span class="nx">fin</span> <span class="p">{</span>
</span><span class='line'>                <span class="c1">// this message is of the current retry level (msg.retries == highWatermark) and the fin flag is set,</span>
</span><span class='line'>                <span class="c1">// meaning this retry level is done and we can go down (at least) one level and flush that</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">expectChaser</span> <span class="p">=</span> <span class="kc">false</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">flushRetryBuffers</span><span class="p">()</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">inFlight</span><span class="p">.</span><span class="nx">Done</span><span class="p">()</span> <span class="c1">// this fin is now handled and will be garbage collected</span>
</span><span class='line'>                <span class="k">continue</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 是怎么刷缓存的消息的            </span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">pp</span> <span class="o">*</span><span class="nx">partitionProducer</span><span class="p">)</span> <span class="nx">flushRetryBuffers</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d state change to [flushing-%d]\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">)</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="o">--</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">updateLeader</span><span class="p">();</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">pp</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">returnErrors</span><span class="p">(</span><span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">buf</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>                <span class="k">goto</span> <span class="nx">flushDone</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d selected broker %d\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">leader</span><span class="p">.</span><span class="nx">ID</span><span class="p">())</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>        <span class="c1">// 可以看到是一层一层开始刷的</span>
</span><span class='line'>        <span class="c1">// 从最外层（即重试次数最多的消息）开始刷起</span>
</span><span class='line'>        <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">msg</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">buf</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">pp</span><span class="p">.</span><span class="nx">brokerProducer</span><span class="p">.</span><span class="nx">input</span> <span class="o">&lt;-</span> <span class="nx">msg</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">flushDone</span><span class="p">:</span>
</span><span class='line'>        <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">buf</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">retryState</span><span class="p">[</span><span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">].</span><span class="nx">expectChaser</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d state change to [retrying-%d]\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span><span class="p">)</span>
</span><span class='line'>            <span class="k">break</span>
</span><span class='line'>        <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">highWatermark</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;producer/leader/%s/%d state change to [normal]\n&quot;</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">pp</span><span class="p">.</span><span class="nx">partition</span><span class="p">)</span>
</span><span class='line'>            <span class="k">break</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<blockquote><p>同步发送与异步发送的区别？</p></blockquote>

<p>和一般的理解不一样，<code>同步发送</code>并不代表消息是同步发出去的，<code>异步发送</code>也只是提供了一个异步的表象。查看源码可以知道，消息的异步同步与否，最关键的是看<code>RequiredAcks</code>配置的是啥。</p>

<p><code>RequiredAcks</code>有3种选择：</p>

<ul>
<li>0 表示不需要等待kafka server确认   # 完全的异步</li>
<li>1 表示需要等待分区的Leader确认后才可以  # 依然可能丢消息</li>
<li>-1 表示需要等待分区的所有副本都确认后才可以   # 完全的同步</li>
</ul>


<p>也就是说，即便你使用了<code>同步发送</code>，但<code>RequiredAcks</code>配置为0，那么也是可能丢消息的（因为client发送消息时并不关注server的返回）</p>

<p>而若<code>RequiredAcks</code>配置为1或者-1，则不论使用同步还是异步，都会”等待“server端的返回。区别是，<code>同步发送</code>真的会同步等待返回，而<code>异步发送</code>则是给你一个选择，返回是在一个单独的channel里，你感兴趣的话，可以自己读取</p>

<h4>可以借鉴学习的代码</h4>

<p>从以下代码可以学习它的<code>重试</code>机制的实现，在函数内部实现了一个<code>retry</code>函数，在发生不需要retry的 error时函数直接return，当遇到需要重试的error时，直接调用<code>retry</code>，<code>retry</code>里封装了重试的策略以及次数等：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// client.go的tryRefreshMetadata方法</span>
</span><span class='line'><span class="c1">// 作用是刷新 Metadata</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">client</span> <span class="o">*</span><span class="nx">client</span><span class="p">)</span> <span class="nx">tryRefreshMetadata</span><span class="p">(</span><span class="nx">topics</span> <span class="p">[]</span><span class="kt">string</span><span class="p">,</span> <span class="nx">attemptsRemaining</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">deadline</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">pastDeadline</span> <span class="o">:=</span> <span class="kd">func</span><span class="p">(</span><span class="nx">backoff</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Duration</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span> <span class="p">!</span><span class="nx">deadline</span><span class="p">.</span><span class="nx">IsZero</span><span class="p">()</span> <span class="o">&amp;&amp;</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">().</span><span class="nx">Add</span><span class="p">(</span><span class="nx">backoff</span><span class="p">).</span><span class="nx">After</span><span class="p">(</span><span class="nx">deadline</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="c1">// we are past the deadline</span>
</span><span class='line'>            <span class="k">return</span> <span class="kc">true</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="nx">retry</span> <span class="o">:=</span> <span class="kd">func</span><span class="p">(</span><span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">attemptsRemaining</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">backoff</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">computeBackoff</span><span class="p">(</span><span class="nx">attemptsRemaining</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">pastDeadline</span><span class="p">(</span><span class="nx">backoff</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;client/metadata skipping last retries as we would go past the metadata timeout&quot;</span><span class="p">)</span>
</span><span class='line'>                <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;client/metadata retrying after %dms... (%d attempts remaining)\n&quot;</span><span class="p">,</span> <span class="nx">client</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Metadata</span><span class="p">.</span><span class="nx">Retry</span><span class="p">.</span><span class="nx">Backoff</span><span class="o">/</span><span class="nx">time</span><span class="p">.</span><span class="nx">Millisecond</span><span class="p">,</span> <span class="nx">attemptsRemaining</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">backoff</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">time</span><span class="p">.</span><span class="nx">Sleep</span><span class="p">(</span><span class="nx">backoff</span><span class="p">)</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="k">return</span> <span class="nx">client</span><span class="p">.</span><span class="nx">tryRefreshMetadata</span><span class="p">(</span><span class="nx">topics</span><span class="p">,</span> <span class="nx">attemptsRemaining</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="nx">deadline</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">broker</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">any</span><span class="p">()</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">;</span> <span class="nx">broker</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="p">!</span><span class="nx">pastDeadline</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span> <span class="nx">broker</span> <span class="p">=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">any</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>
</span><span class='line'>        <span class="o">...</span><span class="p">.</span>
</span><span class='line'>
</span><span class='line'>        <span class="nx">req</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">MetadataRequest</span><span class="p">{</span><span class="nx">Topics</span><span class="p">:</span> <span class="nx">topics</span><span class="p">,</span> <span class="nx">AllowAutoTopicCreation</span><span class="p">:</span> <span class="nx">allowAutoTopicCreation</span><span class="p">}</span>
</span><span class='line'>        <span class="o">...</span><span class="p">.</span>
</span><span class='line'>        <span class="nx">response</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">GetMetadata</span><span class="p">(</span><span class="nx">req</span><span class="p">)</span>
</span><span class='line'>        <span class="k">switch</span> <span class="nx">err</span><span class="p">.(</span><span class="kd">type</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="kc">nil</span><span class="p">:</span>
</span><span class='line'>            <span class="nx">allKnownMetaData</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">topics</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
</span><span class='line'>            <span class="c1">// valid response, use it</span>
</span><span class='line'>            <span class="nx">shouldRetry</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">updateMetadata</span><span class="p">(</span><span class="nx">response</span><span class="p">,</span> <span class="nx">allKnownMetaData</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">shouldRetry</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;client/metadata found some partitions to be leaderless&quot;</span><span class="p">)</span>
</span><span class='line'>                <span class="k">return</span> <span class="nx">retry</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="c1">// note: err can be nil</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>        <span class="o">...</span><span class="p">.</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">case</span> <span class="nx">KError</span><span class="p">:</span>
</span><span class='line'>            <span class="c1">// if SASL auth error return as this _should_ be a non retryable err for all brokers</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">KError</span><span class="p">)</span> <span class="o">==</span> <span class="nx">ErrSASLAuthenticationFailed</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;client/metadata failed SASL authentication&quot;</span><span class="p">)</span>
</span><span class='line'>                <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="k">if</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">KError</span><span class="p">)</span> <span class="o">==</span> <span class="nx">ErrTopicAuthorizationFailed</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;client is not authorized to access this topic. The topics were: &quot;</span><span class="p">,</span> <span class="nx">topics</span><span class="p">)</span>
</span><span class='line'>                <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>            <span class="c1">// else remove that broker and try again</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;client/metadata got error from broker %d while fetching metadata: %v\n&quot;</span><span class="p">,</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">ID</span><span class="p">(),</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>            <span class="nx">_</span> <span class="p">=</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">client</span><span class="p">.</span><span class="nx">deregisterBroker</span><span class="p">(</span><span class="nx">broker</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">default</span><span class="p">:</span>
</span><span class='line'>            <span class="c1">// some other error, remove that broker and try again</span>
</span><span class='line'>            <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;client/metadata got error from broker %d while fetching metadata: %v\n&quot;</span><span class="p">,</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">ID</span><span class="p">(),</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>            <span class="nx">_</span> <span class="p">=</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">client</span><span class="p">.</span><span class="nx">deregisterBroker</span><span class="p">(</span><span class="nx">broker</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="nx">broker</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">Logger</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;client/metadata not fetching metadata from broker %s as we would go past the metadata timeout\n&quot;</span><span class="p">,</span> <span class="nx">broker</span><span class="p">.</span><span class="nx">addr</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">retry</span><span class="p">(</span><span class="nx">ErrOutOfBrokers</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">Logger</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;client/metadata no available broker to send metadata request to&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="nx">client</span><span class="p">.</span><span class="nx">resurrectDeadBrokers</span><span class="p">()</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">retry</span><span class="p">(</span><span class="nx">ErrOutOfBrokers</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[designing data intensive applications part1]]></title>
    <link href="http://cs50Mu.github.io/blog/2020/11/12/designing-data-intensive-applications-part1/"/>
    <updated>2020-11-12T14:27:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2020/11/12/designing-data-intensive-applications-part1</id>
    <content type="html"><![CDATA[<h2>Part I &ndash; Foundations of Data Systems</h2>

<h3>Chapter 1 &ndash; Reliable, Scalable, and Maintainable Applications</h3>

<blockquote><p>Reliability  / 可用性</p></blockquote>

<p>The system should continue to work correctly (performing the correct function at the desired level of performance) even in the face of adversity (hardware or soft‐ ware faults, and even human error).</p>

<p>Note that a fault is not the same as a failure [2]. A fault is usually defined as one component of the system deviating from its spec, whereas a failure is when the system as a whole stops providing the required service to the user. 注意区分 fault 和 failure</p>

<p>有哪些种类的falut呢？</p>

<ul>
<li>hardware faults：硬盘坏了</li>
<li>software errors：内存泄漏</li>
<li>human errors：运维操作失误</li>
</ul>


<blockquote><p>Scalability / 可扩展性</p></blockquote>

<p>As the system grows (in data volume, traffic volume, or complexity), there should be reasonable ways of dealing with that growth.</p>

<p>Even if a system is working reliably today, that doesn’t mean it will necessarily <strong>work reliably in the future</strong>. One common reason for degradation is <strong>increased load.</strong></p>

<p>Scalability is the term we use to describe <strong>a system’s ability to cope with increased load.</strong></p>

<p>Note, however, that it is not a one-dimensional label that we can attach to a system: <strong>it is meaningless to say “X is scalable” or “Y doesn’t scale.”</strong> Rather, discussing scalability means considering questions like “If the system grows in a particular way, what are our options for coping with the growth?” and “How can we add computing resources to handle the additional load?”</p>

<p><strong>Describing Load</strong></p>

<p>First, we need to succinctly describe the current load on the system; only then can we discuss growth questions. 首先要明确现状</p>

<p><strong>Describing Performance</strong></p>

<p>You can look at it in two ways:</p>

<ul>
<li>When you increase a load parameter and keep the system resources (CPU, mem‐ ory, network bandwidth, etc.) unchanged, how is the performance of your system affected?</li>
<li>When you increase a load parameter, how much do you need to increase the resources if you want to keep performance unchanged?</li>
</ul>


<p><strong>percentile</strong>：For example, if the 95th percentile response time is 1.5 seconds, that means 95 out of 100 requests take less than 1.5 seconds, and 5 out of 100 requests take 1.5 seconds or more.</p>

<p><strong>Approaches for Coping with Load</strong></p>

<p>An architecture that is appropriate for one level of load is unlikely to cope with 10 times that load.</p>

<ul>
<li>vertical scaling, moving to a more powerful machine</li>
<li>horizontal scaling, distributing the load across multiple smaller machines</li>
</ul>


<p>An architecture that scales well for a particular application is built around assumptions of which operations will be common and which will be rare. 没有万能的架构，都是根据场景而来的，不能照搬</p>

<p>In an early-stage startup or an unpro‐ ven product it’s usually more important to be able to iterate quickly on product features than it is to scale to some hypothetical future load. 前期先重点关注feature</p>

<blockquote><p>Maintainability / 可维护性</p></blockquote>

<p>Over time, many different people will work on the system (engineering and oper‐ ations, both maintaining current behavior and adapting the system to new use cases), and they should all be able to work on it <strong>productively</strong></p>

<p><strong>Simplicity: Managing Complexity</strong></p>

<p>accidental complexity: we define complexity as accidental if it is not inherent in the problem that the software solves (as seen by the users) but arises only from the implementation.</p>

<h3>CHAPTER 2 &ndash; Data Models and Query Languages</h3>

<p>each layer hides the complexity of the layers below it by providing a clean data model.</p>

<p><strong>The Object-Relational Mismatch</strong></p>

<p>Most application development today is done in object-oriented programming lan‐ guages, which leads to a common criticism of the SQL data model: if data is stored in relational tables, an awkward translation layer is required between the objects in the application code and the database model of tables, rows, and columns. The discon‐ nect between the models is sometimes called an impedance mismatch</p>

<p>However, when it comes to representing many-to-one and many-to-many relation‐ ships, relational and document databases are not fundamentally different: in both cases, the related item is referenced by a unique identifier, which is called a foreign key in the relational model and a document reference in the document model [9]. <strong>That identifier is resolved at read time by using a join or follow-up queries.</strong> 还是需要join或者再次查询</p>

<p>如何选择？realational model OR document model？</p>

<p>跟场景有关</p>

<p>If the data in your application has a document-like structure (i.e., a tree of one-to-many relationships, where typically the entire tree is loaded at once), then it’s probably a good idea to use a document model.</p>

<p>However, if your application does use many-to-many relationships, the document model becomes less appealing. It’s possible to reduce the need for joins by denormal‐ izing, but then the application code needs to do additional work to keep the denor‐ malized data consistent. Joins can be emulated in application code by making multiple requests to the database, but that also moves complexity into the application and is usually slower than a join performed by specialized code inside the database. In such cases, using a document model can lead to significantly more complex appli‐ cation code and worse performance</p>

<p><strong>Schema flexibility in the document model</strong></p>

<p>Document databases are sometimes called schemaless, but that’s misleading, as the code that reads the data usually assumes some kind of structure—i.e., there is an implicit schema, but it is not enforced by the database. 所谓的schemaless</p>

<h3>Storage and Retrieval</h3>

<p><strong>storage engines</strong></p>

<ul>
<li>log-structured storage engines</li>
<li>page-oriented storage engines, such as B-trees</li>
</ul>


<p>In this book, <strong>log</strong> is used in the more general sense: an <strong>append-only</strong> sequence of records. It doesn’t have to be human-readable; it might be binary and intended only for other programs to read.</p>

<p>索引</p>

<p>An index is an additional structure that is <strong>derived from the primary data</strong>. Many databases allow you to add and remove indexes, and this doesn’t affect the contents of the database; it only affects the performance of queries. Maintaining additional structures incurs overhead, especially on writes.</p>

<p><strong>Hash Indexes</strong></p>

<p>keep an <strong>in-memory hash map</strong> where every key is mapped to a byte offset in the data file—the location at which the value can be found.</p>

<p>In this kind of workload, there are a lot of writes, but there are not too many distinct keys—you have a large number of writes per key, but it’s feasible to keep all keys in memory. 适用场景是distinct key的数量不多，而且大部分操作是针对同一个key的update</p>

<p>局限性</p>

<ul>
<li>The hash table must fit in memory, so if you have a very large number of keys, you’re out of luck. 不适用于有很多key的场景</li>
<li>Range queries are not efficient. 区间查询效率不高，因为只能loop所有的key</li>
</ul>


<p><strong>SSTables and LSM-Trees</strong></p>

<p>SSTable(Sorted String Table): 日志中的记录是按<code>key</code>排序的，且<code>key</code>在每个文件中唯一</p>

<p>它有如下优势：</p>

<ul>
<li>Merging segments is simple and efficient. 因为每个日志文件都是按<code>key</code>排序的，可以利用<code>merge sort</code>中的算法来快速merge多个文件。</li>
<li>In order to find a particular key in the file, you no longer need to keep an index of all the keys in memory. 不需要将所有的<code>key</code>索引在内存里了，可以一个<code>key</code>对应一个块(segment)，这叫sparse index(稀疏索引？)。若要找某个key，可以直接确定它大概在哪个块里，而块里的<code>key</code>都是有序的，搜寻也很快。</li>
</ul>


<p>那么如何创建和维护这个SSTables呢？</p>

<p>很明显，key的写入肯定是乱序的，但我们可以先把它们写入内存，然后按序读出再写入硬盘（红黑树、AVL树等都可以做到），现在我们的算法变成：</p>

<ul>
<li>当要写入一个key的时候，我们先把它写入一个平衡树（红黑树、AVL树等）中，我们一般叫这个平衡树的数据结构为<code>memtable</code></li>
<li>当<code>memtable</code>越来越大超过某个阈值（通常为几M）时，我们把它写入一个<code>SSTable</code>文件中。直接中序遍历这棵树就能得到按序排列的key</li>
<li>当要查找某个key时，我们先在<code>memtable</code>中找，若没有，再从<code>SSTable</code>文件中找</li>
</ul>


<p>这个方案有个问题，当数据库突然挂掉时，在<code>memtable</code>但还没有写入<code>SSTable</code>文件中的这些数据会丢。一个解决办法是：在写入<code>memtable</code>的同时，也写一份到append-only日志里，当然是乱序的，但不要紧，我们只是用它来恢复<code>memtable</code>的。</p>

<p>LSM-Trees： 是 Log-Structured Merge-Tree 的缩写。Storage engines that are based on this principle of merging and compacting sorted files are often called <strong>LSM storage engines</strong>. 感觉跟<code>SSTable</code>是同一种东西的不同叫法，或者说，<code>LSM-Trees</code>是更宽泛的叫法。</p>

<p>Lucene（ES用到的索引引擎）也是用的类似的数据结构来实现的全文搜索。 This is implemented with a key-value structure where the key is a word (a term) and the value is the list of IDs of all the documents that contain the word (the postings list).</p>

<p>性能如何呢？</p>

<p>Even though there are many subtleties, the basic idea of LSM-trees—keeping a cascade of SSTables that are merged in the background—is <strong>simple and effective</strong>. Even when the dataset is much bigger than the available memory it continues to work well. Since data is stored in sorted order, you can <strong>efficiently perform range queries</strong> (scanning all keys above some minimum and up to some maximum), and because the disk writes are sequential the LSM-tree can <strong>support remarkably high write throughput.</strong> 亮点是：支持区间查询、写入时性能高。</p>

<p><strong>B-Trees</strong></p>

<p>与 log-structured indexes 的区别：</p>

<ul>
<li>The log-structured indexes we saw earlier break the database down into <strong>variable-size segments</strong>, typically several megabytes or more in size, and always write a segment <strong>sequentially</strong>.</li>
<li>B-trees break the database down into <strong>fixed-size blocks or pages</strong>, traditionally 4 KB in size (sometimes bigger), and read or write one page at a time. This design corresponds more closely to the underlying hardware, as disks are also arranged in fixed-size blocks.</li>
<li>The basic underlying write operation of a B-tree is to overwrite a page on disk with new data. This is in stark contrast to log-structured indexes such as LSM-trees, which only append to files (and eventually delete obsolete files) but never modify files in place. B-tree 会 in-place update，而 LSM-tree只会 append（这是函数式编程的风格）</li>
</ul>


<p><img src="http://cs50Mu.github.io/images/post/b-tree.jpg" alt="" /></p>

<p>The number of references to child pages in one page of the B-tree is called <strong>the branching factor</strong>. For example, in Figure 3-6 the branching factor is six. In practice, the branching factor depends on the amount of space required to store the page references and the range boundaries, but typically it is <strong>several hundred.</strong> 其实就是这个棵树的某个节点有多少个分叉</p>

<p>如何update某个key / val呢？</p>

<p>从 root 开始，类似二分查找，直到找到包含这个key的page，然后更新即可，注意更新是以<code>page</code>为单位的</p>

<p>如何insert呢？</p>

<p>同样也是先查找，找到需要插入的节点（page）后，插入即可，但这里有一个问题，当要插入的page已经太大了时，需要把它split成两个。</p>

<p>经验：</p>

<p>a B-tree with n keys always has a depth of O(log n). Most databases can fit into a B-tree that is <strong>three or four levels</strong> deep, so you don’t need to follow many page references to find the page you are looking for. (<strong>A four-level tree of 4 KB pages with a branching factor of 500 can store up to 256 TB</strong>.)</p>

<p>当数据库突然挂掉时，B-tree 仍然也会丢数据，解决方案仍然也是万金油：append-only log，不过在这里是叫：write-ahead log（WAL, also known as a redo log）This is an append-only file to which every B-tree modification must be written before it can be applied to the pages of the tree itself. When the database comes back up after a crash, this log is used to restore the B-tree back to a consistent state.</p>

<p><strong>Comparing B-Trees and LSM-Trees</strong></p>

<p>As a rule of thumb, <strong>LSM-trees are typically faster for writes, whereas B-trees are thought to be faster for reads</strong> [23]. Reads are typically slower on LSM-trees because they have to check several different data structures and SSTables at different stages of compaction.</p>

<p>LSM tree 的优势：</p>

<ul>
<li>LSM tree 的写放大（write amplification）相对于b-tree要小</li>
<li>LSM tree 更容易被压缩，且不容易出现文件碎片，磁盘利用率更高</li>
</ul>


<p>LSM tree 的劣势</p>

<ul>
<li>compaction process 会可能会影响到读和写，导致读写性能不稳定，而 B-tree 是很稳定的。</li>
<li>在极端情况下，compaction process 的速度会赶不上记录被写入的速度，导致文件系统被写爆（需要一个监控）</li>
<li>LSM tree 会出现一个 key 存在于多个 segments 文件的情况，而 b-tree 能保证一个key只在一个 index 文件中，因而也更容易锁住它，更容易实现事务。</li>
</ul>


<p><strong>Other Indexing Structures</strong></p>

<blockquote><p>Storing values within the index</p></blockquote>

<p>The key in an index is the thing that queries search for, but the value can be one of two things: <strong>it could be the actual row (document, vertex) in question, or it could be a reference to the row stored elsewhere.</strong> 索引里可以直接放真实的记录，也可以放记录的指针</p>

<p>In the latter case, the place where rows are stored is known as a <strong>heap file</strong>, and it stores data in no particular order. The heap file approach is common because it avoids duplicating data when multiple secondary indexes are present: each index just references a location in the heap file, and the actual data is kept in one place. 这种方式可以节省空间，比如多个二级索引可以复用一个记录指针，真实记录只在 heap 中存一份即可</p>

<p>In some situations, the extra hop from the index to the heap file is too much of a per‐ formance penalty for reads, so it can be desirable to <strong>store the indexed row directly within an index</strong>. This is known as a <strong>clustered index</strong>. 聚簇索引?</p>

<p>A compromise between a clustered index (storing all row data within the index) and a nonclustered index (storing only references to the data within the index) is known as a <strong>covering index or index with included columns</strong>, which stores some of a table’s columns within the index [33]. This allows some queries to be answered by using the index alone. 将部分字段的value一起写在index里，若查询的就是其中的字段则可以直接返回，very smart!</p>

<blockquote><p>Multi-column indexes</p></blockquote>

<p>The most common type of multi-column index is called a concatenated index, which simply combines several fields into one key by appending one column to another (the index definition specifies in which order the fields are concatenated)</p>

<p>注意这跟上面说的 covering index 不太一样，这应该是我们平常理解的<code>联合索引</code>了</p>

<blockquote><p>Keeping everything in memory</p></blockquote>

<p>in-memory databases: Redis / Memcached</p>

<p>Counterintuitively, the performance advantage of in-memory databases is not due to the fact that they don’t need to read from disk. Even a disk-based storage engine may never need to read from disk if you have enough memory, because the operating system caches recently used disk blocks in memory anyway. Rather, they can be faster because they can avoid the overheads of encoding in-memory data structures in a form that can be written to disk. 确实挺反直觉的，原来内存数据库快是快在了“避免了从内存数据结构编码成字节”。一般直觉会认为，传统依赖硬盘的数据库的慢是因为它们要读盘、写盘，但其实由于操作系统的<code>虚拟内存</code>机制，大部分的查询都只在内存就能满足了，无需走硬盘。</p>

<p><strong>online transaction processing (OLTP) vs online analytic processing (OLAP)</strong></p>

<p>前者是在线交易型需求，后者是分析分析型需求</p>

<p>In the early days of business data processing, a write to the database typically corre‐ sponded to a commercial transaction taking place: making a sale, placing an order with a supplier, paying an employee’s salary, etc. As databases expanded into areas that didn’t involve money changing hands, the term transaction nevertheless stuck, <strong>referring to a group of reads and writes that form a logical unit</strong>. <code>transaction</code> 一词的由来以及意思，注意跟数据库里的<code>事务</code>区别</p>

<p>OLTP的使用场景如下：</p>

<p>An application typically looks up a small number of records by some key, using an index. Records are inserted or updated based on the user’s input. Because these applications are interactive, the access pattern became known as online transaction processing (OLTP).</p>

<p>OLAP的使用场景如下：</p>

<p>However, databases also started being increasingly used for data analytics, which has very different access patterns. Usually an analytic query needs to <strong>scan over a huge number of records, only reading a few columns per record</strong>, and calculates aggregate statistics (such as count, sum, or average) rather than returning the raw data to the user.</p>

<p><img src="http://cs50Mu.github.io/images/post/oltp-olap.jpg" alt="" /></p>

<p>These OLTP systems are usually expected to be <strong>highly available</strong> and to process transactions with <strong>low latency</strong>, since they are often critical to the operation of the business. Database administrators therefore closely guard their OLTP databases. They are usually reluctant to let business analysts run ad hoc analytic queries on an OLTP database, since those queries are often expensive, scanning large parts of the dataset, which can harm the performance of concurrently executing transactions. 分析型查询不要在业务库上做</p>

<p><strong>Data Warehousing</strong></p>

<p>数据仓库？</p>

<p>A data warehouse, by contrast, is a separate database that analysts can query to their hearts’ content, without affecting OLTP operations [48]. <strong>The data warehouse contains a read-only copy of the data in all the various OLTP systems in the company</strong>. Data is extracted from OLTP databases (using either a periodic data dump or a continuous stream of updates), transformed into an analysis-friendly schema, cleaned up, and then loaded into the data warehouse. This process of getting data into the warehouse is known as Extract–Transform–Load (ETL)</p>

<p>A big advantage of using a separate data warehouse, rather than querying OLTP sys‐ tems directly for analytics, is that the data warehouse can be optimized for analytic access patterns. 把 olap 数据库独立出来的一个好处是，可以针对 olap 类查询做优化</p>

<blockquote><p>Schemas for Analytics</p></blockquote>

<p>数据仓库中主要用到的数据模型是：star schema(also known as dimensional modeling)，翻译成<code>星型结构</code>比较合适，里面主要分为两种表：fact table 和 dimension tables。感觉也没什么特别，就是一张主表，关联了多个子表。</p>

<p>The name “star schema” comes from the fact that when the table relationships are visualized, the fact table is in the middle, surrounded by its dimension tables; the connections to these tables are like the rays of a star. <code>star schema</code>名字的由来</p>

<p>In a typical data warehouse, tables are often very wide: fact tables often have over 100 columns, sometimes several hundred. 数据仓库中的表的字段数量通常很大，超过100，甚至几百个</p>

<p><strong>Column-Oriented Storage</strong></p>

<p>列式存储，为解决大数据查询而提出的优化方案</p>

<p>分析 olap 的查询场景：</p>

<ul>
<li>通常只查询几个字段，一般不会<code>select *</code></li>
</ul>


<p>In most OLTP databases, storage is laid out in a row-oriented fashion: all the values from one row of a table are stored next to each other. 在大部分 oltp 数据库中， 数据是按行存储的，那么虽然我们只需要几个字段，但查询时需要先将这100多个字段先从硬盘中load进内存，然后再返回这几个需要的字段，就是说不管是 select 几个字段，都得把这一整行先读出来。。</p>

<p>那么所谓<code>列式存储</code>其实就是为了规避这个问题，反其道而行之，将数据按列存储，所有记录的某个字段存为一个文件，那么当 select 某个字段时，直接返回这个文件即可。</p>

<p>The column-oriented storage layout relies on each column file containing the rows in the same order. Thus, if you need to reassemble an entire row, you can take the 23rd entry from each of the individual column files and put them together to form the 23rd row of the table. 但需要注意的是，所有列文件里的记录顺序需要是一致的，要不就没法返回某条完整的记录了。</p>

<blockquote><p>Column Compression</p></blockquote>

<p>列式存储还有一个好处是可以采用<code>列压缩</code></p>

<p>Take a look at the sequences of values for each column in Figure 3-10: they often look <strong>quite repetitive</strong>, which is a good sign for compression. Depending on the data in the column, different compression techniques can be used. One technique that is particularly effective in data warehouses is <strong>bitmap encoding</strong> 重复意味着有压缩空间，而我们日常的数据，对于某个字段来说，它的值通常只有有限的数量。</p>

<p>Often, the number of distinct values in a column is small compared to the number of rows (for example, a retailer may have billions of sales transactions, but only 100,000 distinct products). <strong>We can now take a column with n distinct values and turn it into n separate bitmaps: one bitmap for each distinct value, with one bit for each row. The bit is 1 if the row has that value, and 0 if not.</strong></p>

<p>If n is very small (for example, a country column may have approximately 200 dis‐ tinct values), those bitmaps can be stored with one bit per row. But if n is bigger, there will be a lot of zeros in most of the bitmaps (we say that they are sparse). In that case, the bitmaps can additionally be run-length encoded. 如果 n 很大，导致 bitmap 很稀疏，还可以用<code>run-length encoded</code>来进一步优化</p>

<blockquote><p>Writing to Column-Oriented Storage</p></blockquote>

<p>策略跟 LSM-tree 一样，先写到内存中的balanced tree 中，再定期写入文件</p>

<blockquote><p>Aggregation: Data Cubes and Materialized Views</p></blockquote>

<p>Another aspect of data warehouses that is worth mentioning briefly is materialized aggregates. As discussed earlier, data warehouse queries often involve an aggregate function, such as COUNT, SUM, AVG, MIN, or MAX in SQL. If the same aggregates are used by many different queries, it can be wasteful to crunch through the raw data every time. Why not cache some of the counts or sums that queries use most often? 先算一个临时结果</p>

<p><strong>Summary</strong></p>

<p>Disk seek time is often the bottleneck in OLTP.</p>

<p>Disk bandwidth (not seek time) is often the bottleneck in OLAP</p>

<h3>Chapter 4 &ndash; Encoding and Evolution</h3>

<p>需求改变可能会导致数据模型改变，数据模型改变又往往需要改动代码，而通常我们的代码又不可能在瞬间部署完，两个原因：</p>

<ul>
<li>服务端一般需要灰度部署</li>
<li>客户端是否更新，取决于用户的意愿</li>
</ul>


<p>那么，就意味着我们的代码需要对新老数据结构做兼容，两个方面：</p>

<ul>
<li>Backward compatibility / 新代码需要兼容老结构</li>
<li>Forward compatibility / 老代码需要兼容新结构</li>
</ul>


<p><strong>Formats for Encoding Data</strong></p>

<p>Programs usually work with data in (at least) two different representations:</p>

<ul>
<li>In memory, data is kept in objects, structs, lists, arrays, hash tables, trees, and so on. These data structures are optimized for efficient access and manipulation by the CPU (typically using pointers).</li>
<li>When you want to write data to a file or send it over the network, <strong>you have to encode it as some kind of self-contained sequence of bytes</strong> (for example, a JSON document).</li>
</ul>


<p>Thus, we need some kind of translation between the two representations. The trans‐ lation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling) 哈哈，编码与解码、序列化和反序列化，说的都是一个东西</p>

<blockquote><p>JSON, XML, and Binary Variants</p></blockquote>

<p>JSON distinguishes strings and numbers, but it doesn’t distinguish integers and floating-point numbers, and it doesn’t specify a precision. JSON不能区分整数和浮点数，也不能指定精度</p>

<p>JSON and XML have good support for Unicode character strings (i.e., human- readable text), but they don’t support binary strings (sequences of bytes without a character encoding) 不支持 binary strings，解决方案是 base64</p>

<p>JSON 的二进制变种：MessagePack, BSON, BJSON, UBJSON, BISON, and Smile</p>

<p>The binary encoding is 66 bytes long, which is only a little less than the 81 bytes taken by the textual JSON encoding (with whitespace removed). All the binary encodings of JSON are similar in this regard. It’s not clear whether such a small space reduction (and perhaps a speedup in parsing) is worth the loss of human-readability. 这是 MessagePack 的情况，可以看到，二进制版本并没有比普通版本减少多少空间占用。</p>

<blockquote><p>Thrift and Protocol Buffers</p></blockquote>

<p>The big difference compared to Figure 4-1 is that there are no field names (userName, favoriteNumber, interests). Instead, the encoded data contains field tags, which are numbers (1, 2, and 3).  跟上面的二进制变体的最大区别是，不需要字段名称，只要一个tag，因为可以提前定义好 schema</p>

<p>当新增和移除字段时，Thrift 和 Protocol Buffers 都可以做到 backward and forward compatibility</p>

<blockquote><p>Avro</p></blockquote>

<p><a href="http://avro.apache.org/docs/current/">Apache Avro Documentation</a></p>

<p><strong>Modes of Dataflow</strong></p>

<blockquote><p>Dataflow Through Databases</p></blockquote>

<p>In a database, the process that writes to the database encodes the data, and the pro‐ cess that reads from the database decodes it. 有意思，这也算一种数据流动的方式</p>

<blockquote><p>Dataflow Through Services: REST and RPC</p></blockquote>

<p>解惑几个名词：SOA、microservices architecture</p>

<p>This way of building applications has traditionally been called a service- oriented architecture (SOA), more recently refined and rebranded as microservices architecture. 哈哈，同样的东西换个说法又出来了</p>

<p><strong>REST vs SOAP</strong></p>

<p>REST is not a protocol, but rather a design philosophy that builds upon the principles of HTTP [34, 35]. It emphasizes simple data formats, using URLs for identifying resources and using HTTP features for cache control, authentication, and content type negotiation. REST 的特点是能借用 http 的就用</p>

<p>By contrast, SOAP is an XML-based protocol for making network API requests.vii Although it is most commonly used over HTTP, it aims to be independent from HTTP and avoids using most HTTP features. Instead, it comes with a sprawling and complex multitude of related standards. SOAP 的特点是能不用 http 的就不用，啥都自己搞</p>

<p><strong>The problems with remote procedure calls (RPCs)</strong></p>

<p>作者认为 RPC 从根本上就是 flawed，它的出发点是好的，想要把通过网络的访问封装成本地调用的效果，但网络请求跟本地请求有很大差异，这是绕不过去的：</p>

<ul>
<li>本地调用的结果是可预测的，要么成功，要么失败。而网络请求是不可预测的，你发的请求或者对方的回复有可能会丢失，这完全不在你的控制范围之内</li>
<li>本地调用的结果是确定的，而网络调用会出现<code>未定状态</code>，比如请求超时，此时客户端可能会比较懵，不知道是否应该重试</li>
<li>本地调用同一个函数的返回耗时，不管调用多少次应该都是基本不变的，而网络请求的返回会因网络情况不同出现很大的差异</li>
<li>本地调用可以直接传指针（引用），而网络请求只能传递内存对象序列化后的字节</li>
<li>网络请求时，client 和 server 可能是用不同编程语言实现的，而不同编程语言的数据类型又不一样，两者转换时也会出问题</li>
</ul>


<p>Custom RPC protocols with a binary encoding format can achieve better perfor‐ mance than something generic like JSON over REST. However, a RESTful API has other significant advantages: it is good for experimentation and debugging (you can simply make requests to it using a web browser or the command-line tool curl, without any code generation or software installation), it is supported by all main‐ stream programming languages and platforms, and there is a vast ecosystem of tools available (servers, caches, load balancers, proxies, firewalls, monitoring, debugging tools, testing tools, etc.) RPC 的有点是效率高点，而 REST 的巨大优势是：测试和 debug 方便，有大量的工具</p>

<p><strong>Message-Passing Dataflow</strong></p>

<p>消息的好处太多了：</p>

<ul>
<li>It can act as a buffer if the recipient is unavailable or overloaded, and thus improve system reliability.</li>
<li>It can automatically redeliver messages to a process that has crashed, and thus prevent messages from being lost.</li>
<li>It avoids the sender needing to know the IP address and port number of the recipient (which is particularly useful in a cloud deployment where virtual machines often come and go).</li>
<li>It allows one message to be sent to several recipients.</li>
<li>It logically decouples the sender from the recipient (the sender just publishes messages and doesn’t care who consumes them).</li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[「译」Asynchronous Python and Databases]]></title>
    <link href="http://cs50Mu.github.io/blog/2020/01/06/asynchronous-python-and-databases/"/>
    <updated>2020-01-06T15:33:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2020/01/06/asynchronous-python-and-databases</id>
    <content type="html"><![CDATA[<p>翻译一篇 SQLAlchemy 作者的博文，原文：<a href="https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/">Asynchronous Python and Databases</a></p>

<p>这篇文章给异步IO泼了点冷水，引导我们用正确的态度来看待这个技术(其实是任何技术)，不要过度“神化”它，我受益匪浅。</p>

<p>===================== 译文开始 =======================</p>

<p>异步编程的话题很难用几句话来说清。这些年来，这个话题也越来越复杂，而我基本也没有涉猎这个领域。但由于我在Python领域做了很多与关系型数据库交互相关的工作，经常要回答很多有关异步IO和数据库编程的问题，比如跟SQLAlchemy或者Openstack相关的。</p>

<p>不想看长篇大论的，我先简单说一下我的想法：我认为Python的asyncio库设计的非常巧妙、很有前途、使用起来也很有意思。它的代码组织的也足够好，让SQLAlchemy在一定程度上兼容它是可行的。鉴于asyncio已经进入Python标准库，我有兴趣在未来将它引入SQLAlchemy。</p>

<p>尽管我上面说的，我还是认为异步编程只是一种潜在的可用途径，我们不应该把它当成一种万能药，遇到问题就想通过它来解决，除非我们正在写一个HTTP或者聊天服务器（在这种情况下，你需要同时维持大量的慢请求或者空闲TCP连接）。在标准的CRUD模式下的数据库交互代码，从来都没必要使用asyncio，而且若使用的话几乎一定会影响性能（相对于同步io和threading模式）。那些为了所谓的“正确”而对asyncio的吹捧论断是经不起推敲的。</p>

<p>把我的观点说清楚后，让我们开始具体阐述吧</p>

<h3>什么是Asynchronous IO？</h3>

<p>Asynchronous IO是一种获取并发效果的方式，它通过允许进程/线程在IO期间可以继续执行（而不是等待IO完成）来实现。为了实现这种效果，IO函数需要被设置成非阻塞模式（socket的默认模式是阻塞的），因而这些IO函数可以在真正的IO操作完成之前就立即返回。它的实现原理是这样的：通常在一个循环中不断调用一个轮询系统（通常依赖操作系统不同而不同，比如linux下的epoll）来查询当前关注的一批文件描述符是否有数据可读、可写的，若有，则处理它们，当处理完成后，执行权又回到了这个事件循环中来查询下一批可读可写的文件描述符，以此往复。</p>

<p>当一堆线程都在等待一个socket返回结果时是很没有效率的，这时候非阻塞IO就发挥出了它的优势。当你需要监听非常多的TCP socket，并且这些socket大部分时间都在“睡觉”或者是很慢的返回 &ndash; 最好的例子就是聊天服务器或者类似的消息系统，在这种场景下，同时会有非常多的连接存在，但同一时间只有很少的连接会有数据交互，当一个连接上有数据出现时，我们称之为一个IO事件。</p>

<p>近几年，asynchronous IO被成功应用于HTTP服务器和应用中。使用asynchronous IO可以让服务器只用单线程就能处理大量的HTTP连接，特别地，慢HTTP客户端再也不会影响其它正常的客户端的请求。非阻塞的web服务器，比如nginx，在实践中已被证明能很好的工作。</p>

<h3>异步IO和脚本语言</h3>

<p>在脚本语言中，异步io编程主要是通过事件循环来实现的，经典用法是设置一个回调函数，当对应的IO请求的数据已经准备好后会调用这个回调函数。由于事件循环可以提供进程/线程调度的功能，在脚本语言中通常没有必要使用线程。实际上，在代码里混合使用多线程阻塞IO编程和事件循环式的非阻塞IO编程会比较尴尬，因为它们使用的是不同的编程范式。</p>

<p>异步io和事件循环的关系，加上它在服务器端编程因能以直观的方式来提供并发的能力，让它在Javascript语言里特别流行。Javascript是一种用于浏览器上的客户端脚本语言。浏览器，像其它图形应用一样，基本上是一个事件机，事件机做的是，对用户发起的事件（按下按钮、移动鼠标等）做出响应。因此，Javascript里使用回调函数是很平常的事情，而且，直到最近它才支持了多线程。</p>

<p>前端开发者们一直以来都很习惯这些客户端的回调用法，他们开始将回调用于网络事件中，一位新的玩家即将登台，这将把已经火热的Javascript推向新的高潮&hellip;</p>

<h3>服务器</h3>

<p>早在Node.js<a href="https://docs.oracle.com/cd/E19957-01/816-6411-10/getstart.htm#1015788">之前</a>就有尝试想将Javascript用于服务端编程。不过，Node.js成功的一个关键原因在于在它发布之时已经有大量的经验丰富的Javascript程序员，并且它也完全拥抱了客户端Javascript开发者已非常熟悉的事件驱动编程范式。</p>

<p>为了推广，Node.js遵循了一个原则：非阻塞IO模式不仅仅应当被用在大量睡眠或慢请求这样的场景上，更应当做为一个事实上的服务器端编程的<a href="https://www.youtube.com/watch?v=bzkRVzciAZg">标准</a>来推行。这意味着任何类型的网络IO都应该使用非阻塞IO的方式，这当然包括数据库连接了 —— 每个进程通常只维持了少量的(10 &ndash; 50)这种连接，而且已经被&#8221;池化&#8221;，因此因TCP创建连接带来的开销影响会很小，而且，对于一个设计良好的数据库来说，一次数据库请求的时间通常是非常快和可预测的 —— 无论从哪个方面来看，数据库连接的场景都不是一开始引入非阻塞IO所要解决的问题，或者说这个场景跟非阻塞IO所要解决的是恰恰相反的。Postgresql数据库在libpq里支持一个异步命令行API，不过这个API主要是为<a href="http://www.postgresql.org/docs/8.3/static/libpq-async.html">图形应用</a>准备的。</p>

<p>node.js已经受益于一个高性能的<a href="https://code.google.com/p/v8/">JIT引擎</a>了，因此尽管在数据库连接的场景中使用非阻塞IO并不“合适”，但用起来效果还可以。</p>

<h3>线程幽灵</h3>

<p>早在node.js将大量的客户端Javascript开发者转变为服务端异步开发者之前，多线程编程模型已经受到一些<a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.html">学术理论家</a>的抱怨：多线程编程容易产生非确定性的程序，而异步编程，由于使用了一种新的编程范式（事件驱动），迅速被用来攻击多线程编程模型的缺点，主要有两点：一个是由于线程的创建和维持的代价比较大，使得它非常不适合需要同时维持成千上万连接的场景；另一个是多线程的代码很难写，且运行结果是非确定性的。在Python世界中，由于GIL一直以来的问题，async模型做为一个“救世主”的形象，相对于其它地方更容易在这里生根发芽。</p>

<h3>How do you like your Spaghetti? / 你有多喜欢你的意大利面？</h3>

<p>node.js的回调风格的代码以及其它异步风格的范式被认为是有问题的，稍微复杂一点的逻辑如果使用回调风格来组织的话会让代码变得非常繁琐和晦涩难懂，这种代码通常被称为回调面条(<a href="https://www.google.com/search?q=node.js+spaghetti+code&amp;ie=utf-8&amp;oe=utf-8#q=node.js+spaghetti">callback spaghetti</a>)。回调风格的代码到底是一坨屎还是一朵花，这个问题曾经是21世纪最大的争论之一，不过很庆幸我不需要参与其中，因为异步社区已一致确认回调风格的代码是前者并且已经采取措施来改进它这方面的问题。</p>

<p>在Python中，有一种方法可以让你在不使用回调的情况下使用异步IO编程模型，那就是由<a href="http://eventlet.net/">eventlet</a>和<a href="http://www.gevent.org/">gevent</a>提供的所谓“隐式异步IO”。它们通过将IO函数设置为非阻塞模式，启动一批并发的协程（<a href="http://en.wikipedia.org/wiki/Green_threads">green threads</a>），然后依赖底层的事件驱动库（比如<a href="http://libev.schmorp.de/">libev</a>）来根据IO事件的触发来调度这批协程工作。“隐式异步IO”的好处是你原来的同步代码基本是不用动的，只需稍微改动几行代码就可以将你的程序切换到异步模式（当然，肯定是有一些特殊情况造成的小坑，这个另当别论）</p>

<p>与隐式异步IO相对的是，非常有前途的由Python官方提供的异步库<a href="https://www.python.org/dev/peps/pep-3156/">asyncio</a>（本文一开始提到过），现在已进入Python 3标准库了。asyncio库给Python带来了标准的“futures”和<a href="http://en.wikipedia.org/wiki/Coroutine">协程</a>的概念，通过它们我们可以写出跟传统的阻塞代码类似的非阻塞的代码，而在那些需要非阻塞IO操作时会明显区分出来（而不是像gevent/eventlet一样无法区分）。</p>

<h3>SQLAlchemy? Asyncio? 确定吗？</h3>

<p>现在既然asyncio已经进入Python标准库了，是该整合一下之前的老代码了。因为asyncio依旧兼容着之前的返回值和异常机制，跑起来一个asyncio版本的SQLAlchemy没那么难，可能需要几个外部模块使用async results重新实现一下SQLAlchemy Core和ORM的关键方法，大部分代码都不用动。这不再意味着SQLAlchemy的完全重写，并且关于异步的模块可能会独立于主逻辑（并不会污染主逻辑）。整个过程会比较麻烦但 是可行的。</p>

<p>不过，你真的想要一个异步模式的SQLAlchemy吗？</p>

<h3>Taking Async Web Scale</h3>

<p>下面让我们来具体说说，nodejs和asyncio到底哪里做错了，尤其是在与数据库交互时所犯的错误</p>

<h4>问题一 —— 作为性能上魔术仙尘的异步范式</h4>

<p>许多（当然不是全部）node.js社区和Python社区的人声称异步编程范式在几乎所有场景下的并发性能都是很好的。特别地，有一种声音认为显式异步的模式，比如asyncio，的“上下文切换”的代价非常小，再加上Python的那个”臭名昭著“的GIL，他们认为asyncio一定会比使用线程并发的模式更快 —— 至少不会慢。因此，所有的web应用应当尽快换成异步的模式，而且是所有的地方都换，从HTTP请求到数据库请求，这样的话没有任何代价就能提升服务器性能。</p>

<p>我下面仅仅针对数据库请求的情况来阐述，对于HTTP server或者 chat server，使用asyncio可能效果会很好，因为它可以同时维持很多慢连接，但对于本地数据库请求，情况不是这样的。</p>

<h5>1. Python相对于你的数据库来说，非常、<del>非常</del>慢</h5>

<p>更新：Reddit上的Riddlerforce发现了这一节的论述有点问题，即：我不是基于网络连接来测试的，我已经基于网络连接重新做了测试并将结果做了更新，结论是一样的，只是不像一开始那样夸张。</p>

<p>让我们先看一下异步编程有很大<a href="http://blog.kgriffs.com/2012/09/18/demystifying-async-io.html">优势</a>的场景，<a href="http://en.wikipedia.org/wiki/I/O_bound">I/O密集型</a>应用：</p>

<blockquote><p>I/O密集指的是这样一种场景，在这种场景下，由于等待输入输出操作完成所耗费的时间占了整个计算时间的大部分。</p></blockquote>

<p>经常遇到的一个很大的误解是：在一个主要与数据库交互的Python应用中，大部分时间是花费在了它跟数据库的通信上。这个观点在编译型语言上，比如C甚至Java，或许成立，但在Python里不成立。Python 相对于上述编译语言非常慢，尽管PyPy的速度是一个很大的进步了，在简单的CRUD场景（意思是非OLAP类型的重查询，并且网络延迟也很低的情况）下，Python的速度仍然无法跟数据库相提并论。在我为Openstack所做的针对PyMySQL的<a href="https://wiki.openstack.org/wiki/PyMySQL_evaluation">评估</a>中可以看到，纯C写的数据库驱动和纯Python写的驱动，它们的速度相差很大，仅仅考虑驱动的话，Python写的驱动比C写的慢一个数量级。尽管在网络上的耗费会使得CPU和IO的比例更平均，但由Python数据库驱动所耗费的CPU时间仍然比网络IO要多一倍，这还是在没有任何其它数据库抽象层、业务逻辑和呈现逻辑的情况下。</p>

<p>这个<a href="https://techspot.zzzeek.org/files/2015/mysql_speed_test.py">脚本</a>，由Openstack的入口代码改造而来，发起了一堆INSERT和SELECT请求数据库驱动，除此之外基本没有别的Python代码</p>

<p>MySQL-Python，一个纯C写的数据库驱动，在走网络的情况下，有如下表现：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>DBAPI (cProfile):  &lt;module 'MySQLdb'&gt;
</span><span class='line'>     47503 function calls in 14.863 seconds
</span><span class='line'>DBAPI (straight time):  &lt;module 'MySQLdb'&gt;, total seconds 12.962214</span></code></pre></td></tr></table></div></figure>


<p>PyMySQL，一个纯Python写的数据库驱动，大概要慢30%：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>DBAPI (cProfile):  &lt;module 'pymysql'&gt;
</span><span class='line'>     23807673 function calls in 21.269 seconds
</span><span class='line'>DBAPI (straight time):  &lt;module 'pymysql'&gt;, total seconds 17.699732</span></code></pre></td></tr></table></div></figure>


<p>如果不走网络的话（本地直连），PyMySQL要比MySQLdb慢一个数量级：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>DBAPI:  &lt;module 'pymysql'&gt;, total seconds 9.121727
</span><span class='line'>
</span><span class='line'>DBAPI:  &lt;module 'MySQLdb'&gt;, total seconds 1.025674</span></code></pre></td></tr></table></div></figure>


<p>为了看清楚IO操作在整个过程中占多大比例，我们使用<a href="http://www.vrplumber.com/programming/runsnakerun/">RunSnakeRun</a>生成了两张PyMySQL的图，一张是直接请求本地数据库，另一张是走网络请求数据库。这个比例在走网络的时候没本地直连那么夸张，但即使走网络的情况下IO操作也仅占总时间的三分之一，其它三分之二的时间都花在Python处理sql执行返回的结果上。而且，请记住，这仅仅是数据库驱动，一个真实的应用还会有数据库抽象层、业务逻辑和其它展示逻辑。</p>

<p><img src="https://i.imgur.com/arlwrwD.png" alt="" /></p>

<p>本地连接 &ndash; 明显不是IO密集的</p>

<p><img src="https://i.imgur.com/rqoSRP4.png" alt="" /></p>

<p>走网络 &ndash; 没走本地连接时那么夸张，但仍然不是IO密集的（24秒的总执行时间中，操作socket的时间只占8.7秒）</p>

<p>让我们总结一下，在Python里，请求数据库的操作不是一个IO密集型操作（除非你在做大量复杂的查询或者查询会返回很大的数据量，而在高性能应用中肯定要避免这种情况；又或者除非你的应用处在一个很慢的网络环境中）。当我们跟数据库交互时，几乎总是会使用某种类型的连接池，所以创建连接的额外开销已经大大缓解。在正常的网络环境下，数据库的写入和查询速度是很快的。Python本身的开销，仅仅是通过网络封送消息并生成结果集，就给CPU带来了很多工作量，这抵消了非阻塞IO所具有的任何独特的吞吐量优势。 真实应用围绕数据库操作还有大量其它操作，花在CPU上的比例只会增加。</p>

<h5>2. AsyncIO使用了吸引人的但相对来说低效的Python范式</h5>

<p>asyncio的亮点是引入了<code>@asyncio.coroutine</code>这个装饰器，它的作用是将看似同步的代码逻辑推迟到其它协程中执行。这个实现的关键之处是引入了一个新的语法形式：<code>yield from</code>，它的作用是会暂时将函数的执行停止在那一点，将执行权交给其它协程执行，直到事件循环再次将执行权交给它后再次<strong>继续执行</strong>。这是一个很棒的创意，而且它其实用Python已有的<code>yield</code>语句也可以做到，不过用<code>yield from</code>可以让你继续写return语句，这样可以跟之前的非协程代码保持相对的一致。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="nd">@asyncio.coroutine</span>
</span><span class='line'><span class="k">def</span> <span class="nf">some_coroutine</span><span class="p">():</span>
</span><span class='line'>    <span class="n">conn</span> <span class="o">=</span> <span class="k">yield from</span> <span class="n">db</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">conn</span>
</span></code></pre></td></tr></table></div></figure>


<p>这个语法非常好，我很喜欢，不过它增加了函数调用的额外开销，我曾在twitter上<a href="https://twitter.com/zzzeek/status/563865362996133889">发过</a>一个简单的demo来说明这个问题，下面就是这段demo：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="sd">&quot;&quot;&quot;One function calls another normal function, which returns a value.&quot;&quot;&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
</span><span class='line'>    <span class="k">return</span> <span class="mi">5</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">bar</span><span class="p">():</span>
</span><span class='line'>    <span class="n">f1</span> <span class="o">=</span> <span class="n">foo</span><span class="p">()</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">f1</span>
</span><span class='line'>
</span><span class='line'><span class="k">return</span> <span class="n">bar</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">return_with_generator</span><span class="p">():</span>
</span><span class='line'><span class="sd">&quot;&quot;&quot;One function calls another coroutine-like function,</span>
</span><span class='line'><span class="sd">which returns a value.&quot;&quot;&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">decorate_to_return</span><span class="p">(</span><span class="n">fn</span><span class="p">):</span>
</span><span class='line'>    <span class="k">def</span> <span class="nf">decorate</span><span class="p">():</span>
</span><span class='line'>        <span class="n">it</span> <span class="o">=</span> <span class="n">fn</span><span class="p">()</span>
</span><span class='line'>        <span class="k">try</span><span class="p">:</span>
</span><span class='line'>            <span class="n">x</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">it</span><span class="p">)</span>
</span><span class='line'>        <span class="k">except</span> <span class="ne">StopIteration</span> <span class="k">as</span> <span class="n">y</span><span class="p">:</span>
</span><span class='line'>            <span class="k">return</span> <span class="n">y</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">decorate</span>
</span><span class='line'>
</span><span class='line'><span class="nd">@decorate_to_return</span>
</span><span class='line'><span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
</span><span class='line'>    <span class="k">yield from</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>    <span class="k">return</span> <span class="mi">5</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">bar</span><span class="p">():</span>
</span><span class='line'>    <span class="n">f1</span> <span class="o">=</span> <span class="n">foo</span><span class="p">()</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">f1</span>
</span><span class='line'>
</span><span class='line'><span class="k">return</span> <span class="n">bar</span>
</span><span class='line'>
</span><span class='line'><span class="n">return_with_normal</span> <span class="o">=</span> <span class="n">return_with_normal</span><span class="p">()</span>
</span><span class='line'><span class="n">return_with_generator</span> <span class="o">=</span> <span class="n">return_with_generator</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="nn">timeit</span>
</span><span class='line'>
</span><span class='line'><span class="k">print</span><span class="p">(</span><span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">&quot;return_with_generator()&quot;</span><span class="p">,</span>
</span><span class='line'><span class="s">&quot;from __main__ import return_with_generator&quot;</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">10000000</span><span class="p">))</span>
</span><span class='line'><span class="k">print</span><span class="p">(</span><span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">&quot;return_with_normal()&quot;</span><span class="p">,</span>
</span><span class='line'><span class="s">&quot;from __main__ import return_with_normal&quot;</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">10000000</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>结果如下，<code>yield from + StopIteration</code>的版本的耗时是正常的函数调用的6倍左右</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">yield from</span><span class="p">:</span> <span class="mf">12.52761328802444</span>
</span><span class='line'><span class="n">normal</span><span class="p">:</span> <span class="mf">2.110536064952612</span>
</span></code></pre></td></tr></table></div></figure>


<p>针对这个结果，许多人反驳我说，“那又怎么样呢？你的数据库请求的操作占用的时间更多”。但请注意，我们现在不是在讨论一种优化现有代码的方案，而是如何避免将已经很好的代码变得更慢.. PyMySQL的例子应该能说明仅仅在纯Python驱动内，Python自身的额外开销就增加得很快。不过，这个回答可能还是不那么令人信服。</p>

<p>所以，我准备了一个详尽的测试用例，通过这个测试用例我们可以看到Python中的传统线程、asyncio以及gevent形式的异步io这三者之间的对比。我们会使用<a href="http://initd.org/psycopg/"><code>psycopg2</code></a>、<a href="https://github.com/aio-libs/aiopg"><code>aiopg</code></a>和<a href="https://bitbucket.org/dvarrazzo/psycogreen"><code>psycogreen</code></a>。</p>

<p>这个测试用例具体做的是，将几百万条记录尽可能快地插入到Postgresql数据库中，使用上述的三种方式，当然三种方式使用的SQL是完全一样的，这样在保持其它变量一样的情况下，我们就能看出是否是GIL拖慢了我们的程序，又或者asyncio是否能大幅提升我们程序的性能。在这个测试用例中，我没有限制可以同时使用的数据库连接数，在我的测试中，最高曾用到过350个并发连接，如果在实际应用中你这么做的话，相信我，你会被DBA骂死的。</p>

<p>测试的结果放在这个<a href="https://bitbucket.org/zzzeek/bigdata/">README</a>的最后了，包含了在不同的机器、不同的条件下的测试结果。在所有的场景下，性能最好的是当一台机器运行测试代码，去请求运行在另一个机器上的Postgresql数据库时，不过几乎在我运行的每个场景中，不管是在Mac只启动15个线程/协程还是在Linux上启动350个线程/协程，使用线程的代码都比用asyncio要快的多（甚至包括启动350个线程的场景，这挺令我吃惊的），而且使用线程的代码通常也会比使用gevent要快一些（虽然没有asyncio那么夸张）。以下是在我的Linux笔记本上运行120个线程/协程/数据库连接请求另一台Mac笔记本上的Postgresql数据库的测试结果：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">Python2</span><span class="o">.</span><span class="mf">7.8</span> <span class="n">threads</span> <span class="p">(</span><span class="mi">22</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">,</span> <span class="mi">22</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">)</span>
</span><span class='line'><span class="n">Python3</span><span class="o">.</span><span class="mf">4.1</span> <span class="n">threads</span> <span class="p">(</span><span class="mi">10</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">,</span> <span class="mi">21</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">)</span>
</span><span class='line'><span class="n">Python2</span><span class="o">.</span><span class="mf">7.8</span> <span class="n">gevent</span> <span class="p">(</span><span class="mi">18</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">,</span> <span class="mi">19</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">)</span>
</span><span class='line'><span class="n">Python3</span><span class="o">.</span><span class="mf">4.1</span> <span class="n">asyncio</span> <span class="p">(</span><span class="mi">8</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">,</span> <span class="mi">10</span><span class="n">k</span> <span class="n">r</span><span class="o">/</span><span class="n">sec</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>从上面的数据可以看出，对于第一个测试项（第一列）使用asyncio的代码比其它方式要慢多了（Python 3.4 在线程和协程似乎都有点问题），对于第二个测试项（第二列），asyncio的方式也比Python 2.7和Python 3.4下的线程方式要慢一倍。甚至当启动350个并发连接时（通常我们不会在一个单核CPU上启动这么多线程）asyncio的效率也不如线程。即使是使用非常快的、纯C写的psycopg2驱动，仅aiopg库的额外开销以及使用psycopg2的异步库在Python中接收轮询结果的动作，都足以拖慢脚本运行速度。</p>

<p>记住，我甚至没有试图要证明asyncio要比线程慢很多，而只是想说明asyncio并没有（像某些人说的）那样比线程快。跑出来的这个测试结果展示出来的差异比我预期的要大的多。我们也看到了，gevent作为一种相当高效的异步io的方式，还是比线程要慢一些（但没慢太多），这也证实了异步IO范式在这种场景中并不是必然比线程快的，而且从asyncio要比gevent慢很多这个结果可以看出，正是由于asyncio以及其它Python结构的开销再加上本来效率就不高的基于IO的上下文切换机制拖慢了asyncio的性能。</p>

<h4>问题二 —— “异步范式会让写代码更简单”</h4>

<p>这是“魔术仙尘”硬币的另一面。这一论断在“线程是不好”的断言之上更进一步，它指出，如果一个程序用到了线程，比如，你写了一个WSGI web应用，而碰巧又使用线程池的形式运行在<code>mod_wsgi</code>下，那你就是在做“多线程编程”，危险程度不亚于这个<a href="http://www.cs.cmu.edu/afs/cs/academic/class/15492-f07/www/pthreads.html#PITFALLS">多线程编程练习</a>，完全无视了这个事实：你的WSGI应用根本不应该有一点进程共享的逻辑。但不不，你就是在做多线程编程，线程是不好的，你应该立马停止。</p>

<p>“线程是不好的”的论断有一个有趣的转折（哈！），它被显式异步的拥护者用来批判隐式异步技术（比如gevent）。Glyph的这篇<a href="https://glyph.twistedmatrix.com/2014/02/unyielding.html">帖子</a>很好的印证了这一点。他们的论证思路是这样的，如果同意“多线程编程是不好的”这一观点，那么使用隐式模式的异步编程同样也是不好的，因为不管怎么说，隐式模式异步的代码看起来跟多线程编程一样，而且因为IO操作可能发生在代码的任何地方，这种模式跟多线程编程一样是非确定性的。我碰巧也同意这一点，是的，gevent类的协程并发机制并没有比多线程好，不比它差就不错了。一个原因是，Python多线程编程中出现的并发问题比较“轻微”，因为有GIL（尽管我们非常不喜欢它），它让各种非常危险的操作，比如向列表中添加元素，变得安全。但用协程的话，你可以很容易地启动成百上千的协程，也很容易遇到通常在有GIL保护下多线程编程碰不到的<a href="https://github.com/PyMySQL/PyMySQL/issues/275">奇怪问题</a>。</p>

<p>顺便说一句，Glyph给了“魔术仙尘”派一巴掌：</p>

<blockquote><p>不幸的是，经常有人在安利“异步系统”时过分强调异步系统因某种可疑的优化因而能获得相对于多线程模式更好的并发能力，却忽略了我上面谈到的多线程编程模型本身存在的问题。这么来看待“异步”的话，也就不难理解他们会把所有4个选项都放在一起了。</p>

<p>这些年来我一直很内疚，内疚在之前说过的一些话，比如之前我说：一个系统如果使用Twisted实现会比使用线程的性能更好。在很多方面，这个话没错，不过：</p>

<ol>
<li><p>上面只是一个理解猜测，现实中优化性能是一个复杂的工程，涉及到方方面面，</p></li>
<li><p>在真实应用中，“上下文切换”基本不会成为瓶颈，</p></li>
<li><p>上面的话“避重就轻”，其实事件驱动编程的最大优势提现在写更大体量的应用程序上，这个“体量”既指的是项目的代码量大，也指并发使用的用户量大。</p></li>
</ol>
</blockquote>

<p>人们在说到使用asyncio会让你的程序里的bug更少时会援引Glyph的这篇帖子，但同时又会承诺使用asyncio也会带来更高的性能（因“某种原因”选择性忽视Glyph这篇帖子中的某些内容）</p>

<p>Glyph对他的两个观点做了清晰、完美的阐述，即我们既应该用非阻塞IO而且应该用显式地使用它。不过，他的论证过程跟异步IO最初想解决的问题（处理大量并发慢连接的场景）没有一点关系，相反，他说的是事件循环的本质以及这一新的并发模型（避免了把系统级的上下文切换直接暴露给用户）是如何出现的。</p>

<p>我们写了那么久的回调代码，现在使用asyncio终于又可以写正常一点的代码了，这种方式应当仍然需要程序员在代码中明确指出哪些函数会发生IO操作，让我们举一个例子：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">def</span> <span class="nf">transfer</span><span class="p">(</span><span class="n">amount</span><span class="p">,</span> <span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">,</span> <span class="n">server</span><span class="p">):</span>
</span><span class='line'>    <span class="k">if</span> <span class="ow">not</span> <span class="n">payer</span><span class="o">.</span><span class="n">sufficient_funds_for_withdrawal</span><span class="p">(</span><span class="n">amount</span><span class="p">):</span>
</span><span class='line'>        <span class="k">raise</span> <span class="n">InsufficientFunds</span><span class="p">()</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} has sufficient funds.&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>    <span class="n">payee</span><span class="o">.</span><span class="n">deposit</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payee} received payment&quot;</span><span class="p">,</span> <span class="n">payee</span><span class="o">=</span><span class="n">payee</span><span class="p">)</span>
</span><span class='line'>    <span class="n">payer</span><span class="o">.</span><span class="n">withdraw</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} made payment&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>    <span class="n">server</span><span class="o">.</span><span class="n">update_balances</span><span class="p">([</span><span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>从多线程的角度来看，这段代码存在并发的问题：当两个线程同时执行<code>transfer</code>函数时，它们可能都从付款人账户扣款成功而不会报错，从而造成多扣的问题</p>

<p>对应的显式异步的版本如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="nd">@coroutine</span>
</span><span class='line'><span class="k">def</span> <span class="nf">transfer</span><span class="p">(</span><span class="n">amount</span><span class="p">,</span> <span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">,</span> <span class="n">server</span><span class="p">):</span>
</span><span class='line'>    <span class="k">if</span> <span class="ow">not</span> <span class="n">payer</span><span class="o">.</span><span class="n">sufficient_funds_for_withdrawal</span><span class="p">(</span><span class="n">amount</span><span class="p">):</span>
</span><span class='line'>        <span class="k">raise</span> <span class="n">InsufficientFunds</span><span class="p">()</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} has sufficient funds.&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>    <span class="n">payee</span><span class="o">.</span><span class="n">deposit</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payee} received payment&quot;</span><span class="p">,</span> <span class="n">payee</span><span class="o">=</span><span class="n">payee</span><span class="p">)</span>
</span><span class='line'>    <span class="n">payer</span><span class="o">.</span><span class="n">withdraw</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>    <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} made payment&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>    <span class="k">yield from</span> <span class="n">server</span><span class="o">.</span><span class="n">update_balances</span><span class="p">([</span><span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>现在，在最后有一个<code>yield from</code>，那我们知道只有执行到最后那一句的时候程序的执行权才会切换到其它协程，并发执行<code>payer.withdraw()</code>的两个协程，一个协程在执行到最后一句之前，另一个协程没有机会执行。</p>

<p>然后他据此提出为啥隐式异步的模式（gevent）也是不够的，从上面的代码我们可以看出，因为<code>payee.deposit()</code>和<code>payer.withdraw()</code>没有做<code>yield from</code>，因此我们确定在未来的迭代中这两个函数中不会出现IO操作，因而也不会出现并发调用<code>transfer()</code>的情况。</p>

<p>（说个题外话，其实我不太明白，从“我们必须写下<code>yield from</code>，这样我们才知道我们正在干什么”这个角度来说，为什么我们非得需要一个真正的、结构上的程序语言结构（<code>yield from</code>），而不是，比如，一个代码注释，然后这个注释可以被一个兼容gevent/eventlet协程的语法检查器(linter)所识别，然后这个<code>yield from</code>的事情就可以通过语法检查器来完成，而不用新增一个语法，这样的话，既不会对周边的三方库造成影响，也不会产生因显式异步而带来的额外开销。当然，这是另一个话题了。）</p>

<p>抛开显式异步的风格问题，它还有以下两个缺点：</p>

<p>一个是，asyncio让你可以很容易的写<code>yield from</code>语句，这使得它所声称的“可以让你少犯错”的好处大打折扣。Hacker News上的一个读者针对“显式异步的代码更容易debug”的论断做了如下评论：</p>

<blockquote><p>这话的意思基本上是，“我需要在代码中显式表示出上下文切换来，如果不能的话，那这个代码的可读性就非常差”</p>

<p>我认为这是典型的稻草人论证，作者所列举的多线程代码的那些问题是所有重入代码都会出现的问题，不管它是多线程还是单线程。如果你的函数不小心调用了另一个函数，而这个函数又调用了最初的那个函数，那就会出现一样的问题</p>

<p>但是，你猜怎么样，这种情况发生的概率很低，大部分代码不是可重入的，大部分状态不是共享的。</p>

<p>对于那些并发的、有相互调用关系的代码，你必须得认真仔细考虑一下才行，到处写满<code>yield from</code>并不会解决问题。</p>

<p>在实际中，最终你会在你的代码中写满<code>yield from</code>，结果就是你会认为“哈，看起来在哪个地方都有可能切到别的协程执行了”，而这正是你一开始想解决的问题。</p></blockquote>

<p>从我的压测代码中可以看到，这最后一点说的真是没错！下面的代码片段来自多线程版本：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>    <span class="s">&quot;select id from geo_record where fileid=</span><span class="si">%s</span><span class="s"> and logrecno=</span><span class="si">%s</span><span class="s">&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;fileid&#39;</span><span class="p">],</span> <span class="n">item</span><span class="p">[</span><span class="s">&#39;logrecno&#39;</span><span class="p">])</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'><span class="n">row</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
</span><span class='line'><span class="n">geo_record_id</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'><span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>    <span class="s">&quot;select d.id, d.index from dictionary_item as d &quot;</span>
</span><span class='line'>    <span class="s">&quot;join matrix as m on d.matrix_id=m.id where m.segment_id=</span><span class="si">%s</span><span class="s"> &quot;</span>
</span><span class='line'>    <span class="s">&quot;order by m.sortkey, d.index&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;cifsn&#39;</span><span class="p">],)</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'><span class="n">dictionary_ids</span> <span class="o">=</span> <span class="p">[</span>
</span><span class='line'>    <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">cursor</span>
</span><span class='line'><span class="p">]</span>
</span><span class='line'><span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">dictionary_ids</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;items&#39;</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="k">for</span> <span class="n">dictionary_id</span><span class="p">,</span> <span class="n">element</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">dictionary_ids</span><span class="p">,</span> <span class="n">item</span><span class="p">[</span><span class="s">&#39;items&#39;</span><span class="p">]):</span>
</span><span class='line'>    <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>        <span class="s">&quot;insert into data_element &quot;</span>
</span><span class='line'>        <span class="s">&quot;(geo_record_id, dictionary_item_id, value) &quot;</span>
</span><span class='line'>        <span class="s">&quot;values (</span><span class="si">%s</span><span class="s">, </span><span class="si">%s</span><span class="s">, </span><span class="si">%s</span><span class="s">)&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="p">(</span><span class="n">geo_record_id</span><span class="p">,</span> <span class="n">dictionary_id</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span>
</span><span class='line'>    <span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>以下代码来自asyncio版本：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">yield from</span> <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>    <span class="s">&quot;select id from geo_record where fileid=</span><span class="si">%s</span><span class="s"> and logrecno=</span><span class="si">%s</span><span class="s">&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;fileid&#39;</span><span class="p">],</span> <span class="n">item</span><span class="p">[</span><span class="s">&#39;logrecno&#39;</span><span class="p">])</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'><span class="n">row</span> <span class="o">=</span> <span class="k">yield from</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
</span><span class='line'><span class="n">geo_record_id</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'><span class="k">yield from</span> <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>    <span class="s">&quot;select d.id, d.index from dictionary_item as d &quot;</span>
</span><span class='line'>    <span class="s">&quot;join matrix as m on d.matrix_id=m.id where m.segment_id=</span><span class="si">%s</span><span class="s"> &quot;</span>
</span><span class='line'>    <span class="s">&quot;order by m.sortkey, d.index&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;cifsn&#39;</span><span class="p">],)</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'><span class="n">rows</span> <span class="o">=</span> <span class="k">yield from</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
</span><span class='line'><span class="n">dictionary_ids</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'><span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">dictionary_ids</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s">&#39;items&#39;</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="k">for</span> <span class="n">dictionary_id</span><span class="p">,</span> <span class="n">element</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">dictionary_ids</span><span class="p">,</span> <span class="n">item</span><span class="p">[</span><span class="s">&#39;items&#39;</span><span class="p">]):</span>
</span><span class='line'>    <span class="k">yield from</span> <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span><span class='line'>        <span class="s">&quot;insert into data_element &quot;</span>
</span><span class='line'>        <span class="s">&quot;(geo_record_id, dictionary_item_id, value) &quot;</span>
</span><span class='line'>        <span class="s">&quot;values (</span><span class="si">%s</span><span class="s">, </span><span class="si">%s</span><span class="s">, </span><span class="si">%s</span><span class="s">)&quot;</span><span class="p">,</span>
</span><span class='line'>        <span class="p">(</span><span class="n">geo_record_id</span><span class="p">,</span> <span class="n">dictionary_id</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span>
</span><span class='line'>    <span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>注意到它们看起来基本是一样的吗？代码中有没有<code>yield from</code>并没有改变我的代码要表达的意图 —— 这是因为在无聊的与数据库交互的代码中，我们基本上都是在做一些顺序查询。</p>

<p>不管这个想法是否吸引人，实际上它并没多大意义 —— 在程序中使用async或者互斥锁等其它方式来控制并发在任何场景下都是不够的。相反，在真实的无聊的数据库交互代码中，有一个流程是我们绝对总是要做的，那就是：</p>

<h3>通过数据库的ACID来控制并发，而不是通过进程同步机制</h3>

<p>如果我们正在操作的对象是关系型数据库，那不管我们是使用多线程，或是隐式协程，或是显式协程并且用类似<code>yield from</code>标识出了所有可能产生并发竟态的地方，关系都不大。特别是在当今世界，几乎所有的东西都是运行在分布式集群中，那些学院理论学家关于多线程的非确定性的论断只是冰山一角，在分布式集群中并发的情况会发生在完全不同的进程中，非确定性是一定存在的。</p>

<p>在与数据库交互的代码中，你只有一种技术可以用来确保正确的并发，那就是通过数据库提供的事务特性（ACID）。不幸的是，它们并不是“开箱即用”的，尽管有许多出色的工具可以帮助你指引正确的方向。</p>

<p>从数据库角度来说，上面所有的<code>transfer()</code>例子都是错误的。下面是一个正确的版本：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">def</span> <span class="nf">transfer</span><span class="p">(</span><span class="n">amount</span><span class="p">,</span> <span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">,</span> <span class="n">server</span><span class="p">):</span>
</span><span class='line'>    <span class="k">with</span> <span class="n">transaction</span><span class="o">.</span><span class="n">begin</span><span class="p">():</span>
</span><span class='line'>        <span class="k">if</span> <span class="ow">not</span> <span class="n">payer</span><span class="o">.</span><span class="n">sufficient_funds_for_withdrawal</span><span class="p">(</span><span class="n">amount</span><span class="p">,</span> <span class="n">lock</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
</span><span class='line'>            <span class="k">raise</span> <span class="n">InsufficientFunds</span><span class="p">()</span>
</span><span class='line'>        <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} has sufficient funds.&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>        <span class="n">payee</span><span class="o">.</span><span class="n">deposit</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>        <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payee} received payment&quot;</span><span class="p">,</span> <span class="n">payee</span><span class="o">=</span><span class="n">payee</span><span class="p">)</span>
</span><span class='line'>        <span class="n">payer</span><span class="o">.</span><span class="n">withdraw</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span><span class='line'>        <span class="n">log</span><span class="p">(</span><span class="s">&quot;{payer} made payment&quot;</span><span class="p">,</span> <span class="n">payer</span><span class="o">=</span><span class="n">payer</span><span class="p">)</span>
</span><span class='line'>        <span class="n">server</span><span class="o">.</span><span class="n">update_balances</span><span class="p">([</span><span class="n">payer</span><span class="p">,</span> <span class="n">payee</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>


<p>看到区别了吗？在上面的代码中，我们使用了事务。先通过<code>SELECT</code>查询付款人的账户余额然后再使用自动提交来修改他们的余额的做法是完全错误的。我们必须确保我们使用了某种锁机制来获取这个值，因而从我们读取这个值到我们修改它这期间，其它进程不可能基于一个“旧”值来做修改。我们可能使用<code>SELECT .. FOR UPDATE</code>来锁住我们想要修改的这一行记录；或者我们可能使用“读提交”的隔离级别结合一个版本计数器来做一个乐观锁的方案。但无论我们采取哪种方案，这都跟我们要采取的单进程的并发机制（多线程、协程等）都没有关系，因为我们这里说的并发问题涉及的是完全独立的进程之间的交互。</p>

<h3>总结！</h3>

<p>请注意，我说这些不是为了让你不要用asyncio。我认为这个库写的很好，也非常好用，也非常有兴趣把它整合进SQLAlchemy，因为我相信不管别人怎么说大家还是想要asyncio版本的SQLAlchemy&hellip;</p>

<p>我的观点是，当与数据库交互时，使用asyncio相对于传统的多线程范式并没有优势，而且你可能还会观察到一个小到中量的性能上的下降（而不是提升）。这是我的许多同事所熟知的，但是最近我仍然不得不争论这一点。</p>

<p>如果既想在接收web请求时利用非阻塞io的优势，又不想在业务逻辑里写满显式的<code>yield from</code>，那么可以考虑结合使用nginx和uWsgi。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[异步IO的威力]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/11/19/the-power-of-asyncio/"/>
    <updated>2019-11-19T20:12:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/11/19/the-power-of-asyncio</id>
    <content type="html"><![CDATA[<h3>what</h3>

<p>so, what&rsquo;s the problem?</p>

<ul>
<li>压测目标有提升</li>
<li>之前的压测没有考虑三方服务慢返回(通常为200ms左右)的问题，如果当前的压测仍用同步worker会死的佷惨</li>
</ul>


<h3>how</h3>

<p>结合项目的实际情况，问题转化为如何在<code>Python + Django 1.8 + Gunicorn</code>这一组合下使用异步IO。</p>

<p>首先，需要让Gunicorn使用异步worker：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>gunicorn demo.wsgi --workers 3 -k gevent
</span></code></pre></td></tr></table></div></figure>


<p>其次，安装异步worker所需的依赖以及使用与gevent兼容的mysql client库：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'><span class="c"># cat requirements.txt</span>
</span><span class='line'>
</span><span class='line'>...
</span><span class='line'><span class="nv">gevent</span><span class="o">==</span>1.4.0
</span><span class='line'><span class="nv">PyMySQL</span><span class="o">==</span>0.9.3
</span><span class='line'>...
</span></code></pre></td></tr></table></div></figure>


<p>最后，压测标识的处理。之前项目使用的是进程worker，收到压测请求后，是放在threadLocal中的；现在换用异步worker后，worker的scope不再是进程和线程级别的，而是协程级别的。协程级别的local实现也有很多，我们直接使用了werkzeug的实现替换掉了当前的threadLocal</p>

<p>使用同样的硬件资源情况，在请求外部资源耗时200ms的场景下，结果对比：</p>

<p>使用同步worker</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>Running 20s <span class="nb">test</span> @ http://payment2-asyncio.test.svc.luojilab.dc
</span><span class='line'>  10 threads and 40 connections
</span><span class='line'>  Thread Stats   Avg      Stdev     Max   +/- Stdev
</span><span class='line'>    Latency     1.03s   119.85ms   1.92s    92.64%
</span><span class='line'>    Req/Sec     6.45      5.33    30.00     69.53%
</span><span class='line'>  Latency Distribution
</span><span class='line'>     50%    1.01s
</span><span class='line'>     75%    1.05s
</span><span class='line'>     90%    1.09s
</span><span class='line'>     99%    1.67s
</span><span class='line'>  761 requests in 20.10s, 756.54KB <span class="nb">read</span>
</span><span class='line'>Requests/sec:     37.86
</span><span class='line'>Transfer/sec:     37.64KB
</span></code></pre></td></tr></table></div></figure>


<p>使用gevent worker</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>Running 20s <span class="nb">test</span> @ http://payment2-asyncio.test.svc.luojilab.dc
</span><span class='line'>  10 threads and 40 connections
</span><span class='line'>  Thread Stats   Avg      Stdev     Max   +/- Stdev
</span><span class='line'>    Latency   395.56ms   55.53ms 672.69ms   71.48%
</span><span class='line'>    Req/Sec    10.74      5.78    30.00     62.64%
</span><span class='line'>  Latency Distribution
</span><span class='line'>     50%  387.63ms
</span><span class='line'>     75%  424.52ms
</span><span class='line'>     90%  468.89ms
</span><span class='line'>     99%  564.88ms
</span><span class='line'>  2002 requests in 20.08s, 1.94MB <span class="nb">read</span>
</span><span class='line'>Requests/sec:     99.70
</span><span class='line'>Transfer/sec:     99.12KB
</span></code></pre></td></tr></table></div></figure>


<p>不仅qps有了近3倍的提升，而且耗时也明显好很多了。</p>

<h3>why</h3>

<p>我想从两个方面来阐述：</p>

<ul>
<li>异步IO的底层原理</li>
<li>具体到Python + Gevent + gunicorn的原理</li>
</ul>


<h4>操作系统的I/O模型</h4>

<p>我们首先要区分这几种基本的I/O模型，这样才可能理解后续的问题。</p>

<p>对一个socket的读取过程一般分为两个阶段：</p>

<ul>
<li>等待数据准备好（数据到达网卡以及从网卡buffer复制到内核buffer）</li>
<li>数据从内核buffer复制到应用进程的buffer</li>
</ul>


<p>根据系统调用在两个阶段的不同行为，可以定义下面5种I/O模型：</p>

<blockquote><p>Blocking I/O Model / 阻塞 I/O 模型</p></blockquote>

<p>这是大家最熟悉的 I/O 模型，也是socket的默认行为，在两个阶段都是阻塞的</p>

<p><img src="https://i.imgur.com/uAxGXLe.png" alt="" /></p>

<blockquote><p>Nonblocking I/O Model / 非阻塞 I/O 模型</p></blockquote>

<p>我们可以将socket设置为非阻塞的模式，意思是，告诉操作系统，当应用发起一个I/O请求时，如果这个请求无法立即返回（即数据还没准备好），直接返回一个特定的错误就好了，不要让应用一直等待。</p>

<p>这种模型看似是将主动权放到了应用这边，但也意味着应用需要不停的轮询操作系统来确认数据是否已经准备好了。</p>

<p><img src="https://i.imgur.com/Yq9Kiib.png" alt="" /></p>

<blockquote><p>I/O Multiplexing Model / I/O 多路复用模型</p></blockquote>

<p>先发起一个阻塞的系统调用(select / poll / epoll)，当有数据可读后，它会返回，然后应用继续发起recvfrom调用开始读取数据。这种模型乍一看并没有多少优势，反而还比阻塞的模型多了一次系统调用，但它的优点是可以一次监控多个socket（阻塞模型中一次只能等待一个socket）</p>

<p><img src="https://i.imgur.com/vsCsyoE.png" alt="" /></p>

<blockquote><p>Signal-Driven I/O Model / 信号驱动 I/O 模型</p></blockquote>

<p>在这个模型下，我们可以向操作系统注册一个信号处理器（signal handler），当数据准备好后，操作系统会发送一个信号来通知我们，此时我们可以开始发起recvfrom来读取数据。注意，第一个阶段是非阻塞的，第二个阶段是阻塞的</p>

<p><img src="https://i.imgur.com/CFr5zb1.png" alt="" /></p>

<blockquote><p>Asynchronous I/O Model / 异步I/O模型</p></blockquote>

<p>这个模型下，两个阶段的调用都是阻塞的。应用只需发起一个<code>aio_read</code>的调用，然后等着操作系统的通知即可，如果收到了通知，说明此时数据一定是准备好的。</p>

<p>这种模型看起来是最理想的，但不幸的是并没有多少操作系统支持这种 I/O 模型</p>

<p><img src="https://i.imgur.com/vOkfZam.png" alt="" /></p>

<p>小结：</p>

<p>根据 POSIX 的定义：</p>

<ul>
<li>同步I/O操作(synchronous I/O operation)指本次I/O操作会导致请求方阻塞，直至本次I/O操作完成</li>
<li>异步I/O操作(asynchronous I/O operation)指本次I/O操作不会阻塞请求方</li>
</ul>


<p>上述列出的5种I/O模型，前4种都是同步I/O操作，只有最后一种属于异步I/O操作</p>

<h4>epoll</h4>

<p>epoll是linux平台上的 I/O 多路复用模型的实现中的佼佼者。它的性能很好，可以支持大量的并发连接，现在高性能的webserver一般底层都有epoll的功劳（比如Nginx）。</p>

<p>我们首先来看一下它的基本用法，它对外只暴露了3个接口：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>int epoll_create<span class="o">(</span>int size<span class="o">)</span>；
</span><span class='line'>int epoll_ctl<span class="o">(</span>int epfd, int op, int fd, struct epoll_event *event<span class="o">)</span>；
</span><span class='line'>int epoll_wait<span class="o">(</span>int epfd, struct epoll_event * events, int maxevents, int timeout<span class="o">)</span>;
</span></code></pre></td></tr></table></div></figure>


<p>其中，</p>

<ul>
<li><code>epoll_create</code>用于创建一个epoll实例，后续的操作都是基于它；</li>
<li><code>epoll_ctl</code>用于告诉epoll你想关心哪些文件描述符的哪种行为（比如可写、可读）；

<ul>
<li>epfd即为<code>epoll_create</code>返回的epoll实例</li>
<li>op为要对文件描述符监听类别，分为新增（<code>EPOLL_CTL_ADD</code>）、删除（<code>EPOLL_CTL_DEL</code>）和修改（<code>EPOLL_CTL_MOD</code>）</li>
<li>fd即想要监听的文件描述符</li>
<li><code>epoll_event</code>主要用于表示想关注这个fd的哪些状态，比如可写、可读等</li>
</ul>
</li>
<li><code>epoll_wait</code>是等待I/O事件发生，它是一个阻塞请求，但它只要返回了文件描述符，那这些文件描述符就一定是ready的（即数据已准备好，可立即读写，不会阻塞）

<ul>
<li>events 是调用方初始化后传入的，如果有ready的文件描述符，epoll会写入到这里，调用方在<code>epoll_wait</code>返回后处理events即可</li>
<li>timeout，等待的时候可以指定一个超时时间，表示最多只等待这么长时间，若此期间没有任何I/O 事件发生也会返回</li>
</ul>
</li>
</ul>


<p>一个简单的使用示例，伪代码如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">epfd</span> <span class="o">=</span> <span class="n">epoll_init1</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
</span><span class='line'><span class="n">event</span><span class="o">.</span><span class="n">events</span> <span class="o">=</span> <span class="n">EPOLLET</span> <span class="o">|</span> <span class="n">EPOLLIN</span><span class="p">;</span>
</span><span class='line'><span class="n">event</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span> <span class="o">=</span> <span class="n">serverfd</span><span class="p">;</span>
</span><span class='line'><span class="n">epoll_ctl</span><span class="p">(</span><span class="n">epfd</span><span class="p">,</span> <span class="n">EPOLL_CTL_ADD</span><span class="p">,</span> <span class="n">serverfd</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">event</span><span class="p">);</span>
</span><span class='line'><span class="o">//</span> <span class="err">主循环</span>
</span><span class='line'><span class="k">while</span><span class="p">(</span><span class="n">true</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="o">//</span> <span class="err">等待事件</span>
</span><span class='line'>    <span class="n">count</span> <span class="o">=</span> <span class="n">epoll_wait</span><span class="p">(</span><span class="n">epfd</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">events</span><span class="p">,</span> <span class="n">MAXEVENTS</span><span class="p">,</span> <span class="n">timeout</span><span class="p">);</span>
</span><span class='line'>    <span class="o">//</span> <span class="err">遍历</span><span class="n">ready</span><span class="err">的事件</span>
</span><span class='line'>    <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">count</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span><span class="p">(</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">events</span> <span class="o">&amp;</span> <span class="n">EPOLLERR</span> <span class="o">||</span> <span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">events</span> <span class="o">&amp;</span> <span class="n">EPOLLHUP</span><span class="p">)</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">处理错误</span>
</span><span class='line'>        <span class="k">if</span><span class="p">(</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span> <span class="o">==</span> <span class="n">serverfd</span><span class="p">)</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">为接入的连接注册事件</span>
</span><span class='line'>        <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">events</span> <span class="o">&amp;</span> <span class="n">EPOLLIN</span><span class="p">)</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">处理可读的缓冲区</span>
</span><span class='line'>            <span class="n">read</span><span class="p">(</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="nb">len</span><span class="p">);</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">向</span><span class="n">epoll</span><span class="err">注册一个该</span><span class="n">fd</span><span class="err">可写的“关注点”</span>
</span><span class='line'>            <span class="n">event</span><span class="o">.</span><span class="n">events</span> <span class="o">=</span> <span class="n">EPOLLET</span> <span class="o">|</span> <span class="n">EPOLLOUT</span><span class="p">;</span>
</span><span class='line'>            <span class="n">event</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span> <span class="o">=</span> <span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span><span class="p">;</span>
</span><span class='line'>            <span class="n">epoll_ctl</span><span class="p">(</span><span class="n">epfd</span><span class="p">,</span> <span class="n">EPOLL_CTL_MOD</span><span class="p">,</span> <span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">event</span><span class="p">);</span>
</span><span class='line'>        <span class="k">else</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">处理可写的缓冲区</span>
</span><span class='line'>            <span class="n">write</span><span class="p">(</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="nb">len</span><span class="p">);</span>
</span><span class='line'>            <span class="o">//</span> <span class="err">后续可以关闭</span><span class="n">fd</span><span class="err">或者</span><span class="n">MOD</span><span class="err">至</span><span class="n">EPOLLOUT</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>值得注意的是，里面有一个loop forever的循环，因为需要不停的通过发起<code>epoll_wait</code>调用来询问操作系统哪些fd是ready状态了，以便进行处理。记住这一点，我们后面还会提到这个循环。</p>

<p>下面简单说一下epoll为什么效率这么高，要想知道epoll为啥快就得先知道它的先驱者们（select
 / poll）为啥慢，有对比才有伤害嘛。</p>

<p> select 和 poll 的实现原理基本是一样的，细节略有差别。它们的基本思想是调用方提供一批文件描述符给它，假设为n个，它帮你遍历一遍，逐个查看一下这个文件描述符是否可写可读，所以它的实现复杂度是O(n)的，连接数少的时候还好，当连接数数量上来的时候，效率就很差了。</p>

<p> 再说说epoll，它在内部维护了一个就绪列表（ready list），代表了当前可读可写的文件描述符集合，因此当应用调用<code>epoll_wait</code>来问当前有哪些文件描述符可用时，它直接返回这个list即可，一般在同一时间可读可写的I/O数量是很少的，因此它的时间复杂度是O(当前可读可写的文件描述符数量)，甚至可以说是O(1)，当然是非常快的。</p>

<p> 我们可以再追问一句，这个就绪列表是如何实现的呢？它是如何能够反映某一个时刻的文件描述符的可用情况的呢？这里就要讲到epoll的另一个函数<code>epoll_ctl</code>，记得我们上面说它的作用是用于告诉epoll我们关心哪些文件描述符的可读可写，guess what？当你在调用这个函数的时候，epoll会向内核注册一个回调函数，当某个我们关心的文件描述符变为可读 / 可写状态时，内核就通过这个回调函数讲这个文件描述符写入就绪列表了，可见它的实现机制是事件驱动的，这也是epoll名字的由来（event poll）</p>

<p>参考：</p>

<ul>
<li><a href="https://studygolang.com/articles/22460">从源码角度看Golang的TCP Socket(epoll)实现</a></li>
<li><a href="https://zhuanlan.zhihu.com/p/87843750?utm_source=wechat_session&amp;utm_medium=social&amp;utm_oi=26858074669056">深入理解IO复用之epoll</a> 非常好</li>
<li><a href="http://gityuan.com/2019/01/06/linux-epoll/">源码解读epoll内核机制</a>  代码讲的6</li>
<li><a href="http://www.hulkdev.com/posts/epoll-io">论epoll的实现</a></li>
<li><a href="http://linbo.github.io/2019/03/01/epoll-fundamental">epoll的那些事</a></li>
<li><a href="http://blog.lucode.net/linux/epoll-tutorial.html">Linux epoll 详解</a></li>
<li><a href="http://luodw.cc/2016/01/24/epoll/">谈谈epoll实现原理</a></li>
</ul>


<h4>event loop（libev / libuv）</h4>

<p>记得我们上面在介绍epoll时说，它的使用一般是通过一个循环（loop），libev就是这样一种事件循环库，它帮用户屏蔽了很多细节，用户只需要告诉它想要关心的文件描述符，然后启动一个事件循环等待回调即可。</p>

<p>来自 libev 官方文档的介绍：</p>

<blockquote><p>Libev is an event loop: you register interest in certain events (such as a file descriptor being readable or a timeout occurring), and it will manage these event sources and provide your program with events.</p>

<p>To do this, it must take more or less <strong>complete control over your process</strong> (or thread) by executing the event loop handler, and will then <strong>communicate events via a callback mechanism</strong>.</p></blockquote>

<p>可以看到，它不仅仅支持I/O事件，更重要的是它支持跨平台使用，支持了 Linux 平台的 epoll 和 BSD 平台上的kqueue，使用它开发的应用可以在无需更改代码的情况下在几个平台之间无缝切换。</p>

<p>参考：</p>

<ul>
<li><a href="https://jin-yang.github.io/post/linux-libev.html">libev 使用简介</a></li>
<li><a href="https://juejin.im/post/5d412865e51d4561e84fcba7">事件驱动异步IO的真正奥秘之Libuv入门</a></li>
<li><a href="https://www.jianshu.com/p/2c78f7ec7c7f">libev 介绍</a></li>
<li><a href="https://www.ibm.com/developerworks/cn/aix/library/au-libev/index.html">使用 libevent 和 libev 提高网络应用性能</a></li>
<li><a href="https://metacpan.org/pod/distribution/EV/libev/ev.pod">libev官方文档</a></li>
</ul>


<h4>协程 / coroutine</h4>

<p>简单的说，协程就是可以暂时中断，之后可以再继续执行的程序。它跟线程、进程是一个概念上的东西，但又有很大的区别：</p>

<ul>
<li>线程的上下文切换成本很高，但协程之间的切换成本很低，轻易可以产生大量的协程</li>
<li>协程的切换必须发生在一个线程内，不能跨线程切换</li>
<li>线程的切换是由操作系统来控制的，不受程序员控制，即它是抢占式的；而协程的切换是由程序员控制的</li>
</ul>


<p>一个简单的例子来了解下协程是个什么东西：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">time</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">consumer</span><span class="p">():</span>
</span><span class='line'>    <span class="n">r</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
</span><span class='line'>    <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
</span><span class='line'>        <span class="n">n</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">r</span>
</span><span class='line'>        <span class="k">if</span> <span class="ow">not</span> <span class="n">n</span><span class="p">:</span>
</span><span class='line'>            <span class="k">return</span>
</span><span class='line'>        <span class="k">print</span><span class="p">(</span><span class="s">&#39;[CONSUMER] Consuming </span><span class="si">%s</span><span class="s">...&#39;</span> <span class="o">%</span> <span class="n">n</span><span class="p">)</span>
</span><span class='line'>        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>        <span class="n">r</span> <span class="o">=</span> <span class="s">&#39;200 OK&#39;</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">produce</span><span class="p">(</span><span class="n">c</span><span class="p">):</span>
</span><span class='line'>    <span class="n">c</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
</span><span class='line'>    <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span>
</span><span class='line'>    <span class="k">while</span> <span class="n">n</span> <span class="o">&lt;</span> <span class="mi">3</span><span class="p">:</span>
</span><span class='line'>        <span class="n">n</span> <span class="o">=</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span>
</span><span class='line'>        <span class="k">print</span><span class="p">(</span><span class="s">&#39;[PRODUCER] Producing </span><span class="si">%s</span><span class="s">...&#39;</span> <span class="o">%</span> <span class="n">n</span><span class="p">)</span>
</span><span class='line'>        <span class="n">r</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
</span><span class='line'>        <span class="k">print</span><span class="p">(</span><span class="s">&#39;[PRODUCER] Consumer return: </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">r</span><span class="p">)</span>
</span><span class='line'>    <span class="n">c</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="k">if</span> <span class="n">__name__</span><span class="o">==</span><span class="s">&#39;__main__&#39;</span><span class="p">:</span>
</span><span class='line'>    <span class="n">c</span> <span class="o">=</span> <span class="n">consumer</span><span class="p">()</span>
</span><span class='line'>    <span class="n">produce</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行结果如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Producing 1...
</span><span class='line'><span class="o">[</span>CONSUMER<span class="o">]</span> Consuming 1...
</span><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Consumer <span class="k">return</span>: 200 OK
</span><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Producing 2...
</span><span class='line'><span class="o">[</span>CONSUMER<span class="o">]</span> Consuming 2...
</span><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Consumer <span class="k">return</span>: 200 OK
</span><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Producing 3...
</span><span class='line'><span class="o">[</span>CONSUMER<span class="o">]</span> Consuming 3...
</span><span class='line'><span class="o">[</span>PRODUCER<span class="o">]</span> Consumer <span class="k">return</span>: 200 OK
</span></code></pre></td></tr></table></div></figure>


<p>可以看到，我们实现了简单的生产者消费者模式，但我们没有使用多线程，而是在一个线程内使用协程的模式实现的，程序的控制权在生产者和消费者之间轮流切换</p>

<p>参考：</p>

<ul>
<li><a href="http://blog.ez2learn.com/2010/07/17/talk-about-coroutine-and-gevent/">淺談coroutine與gevent</a></li>
<li><a href="https://en.wikipedia.org/wiki/Coroutine">Coroutine</a></li>
</ul>


<h4>Greenlet</h4>

<p>Python2 只支持简单的协程机制(semicoroutine)，Greenlet是一个第三方的 Python 协程实现，它实现了完善的协程机制。</p>

<p>来自官方文档的一个简单例子：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">greenlet</span> <span class="kn">import</span> <span class="n">greenlet</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">test1</span><span class="p">():</span>
</span><span class='line'>    <span class="k">print</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span>
</span><span class='line'>    <span class="n">gr2</span><span class="o">.</span><span class="n">switch</span><span class="p">()</span>
</span><span class='line'>    <span class="k">print</span><span class="p">(</span><span class="mi">34</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">test2</span><span class="p">():</span>
</span><span class='line'>    <span class="k">print</span><span class="p">(</span><span class="mi">56</span><span class="p">)</span>
</span><span class='line'>    <span class="n">gr1</span><span class="o">.</span><span class="n">switch</span><span class="p">()</span>
</span><span class='line'>    <span class="k">print</span><span class="p">(</span><span class="mi">78</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="n">gr1</span> <span class="o">=</span> <span class="n">greenlet</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span>
</span><span class='line'><span class="n">gr2</span> <span class="o">=</span> <span class="n">greenlet</span><span class="p">(</span><span class="n">test2</span><span class="p">)</span>
</span><span class='line'><span class="n">gr1</span><span class="o">.</span><span class="n">switch</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># 输出结果</span>
</span><span class='line'><span class="mi">12</span>
</span><span class='line'><span class="mi">56</span>
</span><span class='line'><span class="mi">34</span>
</span><span class='line'><span class="c"># 注意，78并没有机会输出</span>
</span></code></pre></td></tr></table></div></figure>


<p>参考：</p>

<ul>
<li><a href="https://greenlet.readthedocs.io/en/latest/">greenlet: Lightweight concurrent programming</a></li>
</ul>


<h4>gevent</h4>

<p>讲完了上面的铺垫，终于可以说说我们今天的主角了。</p>

<p>来自官网的介绍：</p>

<blockquote><p>gevent is a <strong>coroutine-based</strong> Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev or libuv event loop.</p></blockquote>

<p>它是构建于协程(greenlet)、事件循环(libev / libuv)之上的一个网络库，可以让你用同步的写法来做异步的事情（而不是js中的那种变态的callback hell写法）。那么问题来了，它是怎么做到的呢？</p>

<ul>
<li>基于协程，可以在用户态控制协程的切换（而不是由操作系统强制切换）</li>
<li>基于事件循环，从而能够快速感知I/O事件</li>
<li>基于上述两者将Python标准库中所有涉及到I/O的库全部重写了一遍（所有对外API保持不变，这是关键），然后提供了一个monkey patch方法可以让用户很便利的一键替换</li>
</ul>


<p>那么重写的库的逻辑是什么呢？无非是当遇到I/O需要等待时，就将控制流切换到别的协程上，等I/O准备好后再伺机切换回来，程序的执行模式大概是这个样子的：</p>

<p><img src="https://i.imgur.com/ElFdb46.png" alt="" /></p>

<p>参考：</p>

<ul>
<li><a href="https://sdiehl.github.io/gevent-tutorial/">gevent For the Working Python Developer</a></li>
<li><a href="https://engineering.mixpanel.com/2010/10/29/gevent-the-good-the-bad-the-ugly/">gevent: the Good, the Bad, the Ugly</a></li>
<li><a href="https://zapier.com/engineering/why-zapier-doesnt-use-gevent-yet/">Why Zapier Doesn&rsquo;t Use Gevent (Yet)</a></li>
<li><a href="https://tech.wayfair.com/2018/07/blocking-io-in-gunicorn-gevent-workers/">Blocking IO in Gunicorn Gevent Workers</a></li>
<li><a href="https://blog.mirumee.com/django-fast-part-2-d73a4ecd61f3">Django, fast: part 2</a></li>
<li><a href="https://medium.com/@bfirsh/squeezing-every-drop-of-performance-out-of-a-django-app-on-heroku-4b5b1e5a3d44">Squeezing every drop of performance out of a Django app on Heroku</a></li>
<li><a href="https://www.slideshare.net/mahendram/scaling-django-with-gevent">Scaling Django with gevent</a></li>
</ul>


<h4>gunicorn 的异步 worker</h4>

<p>在计算机科学中有一句“名言”，没有什么问题不能通过增加一层封装来解决的，如果没解决，那就再封装一层&hellip; 所以，欢迎来到我们的最后一层封装，有了它，我们的工作进一步简化成“通过参数指定一个异步worker类型来启动gunicorn”即可享受异步IO的好处。那么，它做了什么呢？</p>

<ul>
<li>帮我们执行了monkey patch替换掉了标准库中的阻塞实现（是的，使用gunicorn后你甚至都不需要自己monkey patch）</li>
<li>启动了一个协程池来处理请求</li>
</ul>


<h4>“题外话”</h4>

<p>话题跟当前主题不太相关，但也是有关系的，我想再多说几句关于使用wrk的一些心得</p>

<blockquote><p>wrk 是什么？</p></blockquote>

<p>首先明确一点，wrk是压力测试工具，其目的是验证出系统支持的最大QPS，不是为了模拟现实流量的。（wrk is not tool for emulate real load from humans, 100 connections doesn&rsquo;t mean 100 users (browsers). No one can push F5 key 10000 times a second. But wrk can send 10000 requests on 1 connection.）</p>

<blockquote><p>压测时，c 和 t 参数各该如何设置？</p></blockquote>

<p>这个问题曾经困扰了我好久.. 在项目的github issues里也充斥了此类疑问。在围观了各路大神的回复后，有如下结论：</p>

<p>t（线程数）应当根据压测机器的配置来设置（比如<code>2*core + 1</code>)，而c（连接数）则根据被压测接口的表现，从一个初始值开始慢慢提升，直至被压测接口的qps无法再提升为止</p>

<blockquote><p>wrk 的机制（原理）？</p></blockquote>

<p>底层也使用了异步IO，不过是借用的Redis实现的event loop，没有使用常用的libev等库。它会启动 t 个线程，每个线程中开 c / t = m 个连接，然后不停的在一个连接中发request、接response。是的，它利用了 http 的 keep alive特性，不会不停的新建和关闭连接。</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Gin源码分析]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/10/27/source-code-of-gin-framework/"/>
    <updated>2019-10-27T21:55:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/10/27/source-code-of-gin-framework</id>
    <content type="html"><![CDATA[<p>github仓库：<a href="https://github.com/gin-gonic/gin">https://github.com/gin-gonic/gin</a></p>

<h3>what</h3>

<p>Gin is a web framework written in Go (Golang). It features a martini-like API with much better performance, up to 40 times faster thanks to httprouter</p>

<h3>how</h3>

<p>从一个例子开始：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kn">package</span> <span class="nx">main</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="s">&quot;github.com/gin-gonic/gin&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">r</span> <span class="o">:=</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">Default</span><span class="p">()</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">GET</span><span class="p">(</span><span class="s">&quot;/ping&quot;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">gin</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">JSON</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">H</span><span class="p">{</span>
</span><span class='line'>          <span class="s">&quot;message&quot;</span><span class="p">:</span> <span class="s">&quot;pong&quot;</span><span class="p">,</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>  <span class="p">})</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">Run</span><span class="p">()</span> <span class="c1">// listen and serve on 0.0.0.0:8080</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>先看看Default()是如何实现的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Default returns an Engine instance with the Logger and Recovery middleware already attached.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="nx">Default</span><span class="p">()</span> <span class="o">*</span><span class="nx">Engine</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">debugPrintWARNINGDefault</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">engine</span> <span class="o">:=</span> <span class="nx">New</span><span class="p">()</span>
</span><span class='line'>      <span class="c1">// 默认创建的engine自带了logger和recovery中间件</span>
</span><span class='line'>      <span class="c1">// 注意这里设置的是全局的中间件，每一个request都会被</span>
</span><span class='line'>      <span class="c1">// 在这里设置的中间件处理</span>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">Use</span><span class="p">(</span><span class="nx">Logger</span><span class="p">(),</span> <span class="nx">Recovery</span><span class="p">())</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">engine</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后我们来看看创建的engine长什么样</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Engine is the framework&#39;s instance, it contains the muxer, middleware and configuration settings.</span>
</span><span class='line'>  <span class="c1">// Create an instance of Engine, by using New() or Default()</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">Engine</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">RouterGroup</span>
</span><span class='line'>      <span class="c1">// 篇幅原因，以下字段省略</span>
</span><span class='line'>      <span class="c1">//...</span>
</span><span class='line'>   <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Use attaches a global middleware to the router. ie. the middleware attached though Use() will be</span>
</span><span class='line'>  <span class="c1">// included in the handlers chain for every single request. Even 404, 405, static files...</span>
</span><span class='line'>  <span class="c1">// For example, this is the right place for a logger or error management middleware.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span><span class="p">)</span> <span class="nx">Use</span><span class="p">(</span><span class="nx">middleware</span> <span class="o">...</span><span class="nx">HandlerFunc</span><span class="p">)</span> <span class="nx">IRoutes</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 底层调用的是RouterGroup的方法</span>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">RouterGroup</span><span class="p">.</span><span class="nx">Use</span><span class="p">(</span><span class="nx">middleware</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 404时的handler</span>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">rebuild404Handlers</span><span class="p">()</span>
</span><span class='line'>      <span class="c1">// 405时的handler</span>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">rebuild405Handlers</span><span class="p">()</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">engine</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'> <span class="c1">// Use adds middleware to the group, see example code in GitHub.</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">group</span> <span class="o">*</span><span class="nx">RouterGroup</span><span class="p">)</span> <span class="nx">Use</span><span class="p">(</span><span class="nx">middleware</span> <span class="o">...</span><span class="nx">HandlerFunc</span><span class="p">)</span> <span class="nx">IRoutes</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">group</span><span class="p">.</span><span class="nx">Handlers</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">group</span><span class="p">.</span><span class="nx">Handlers</span><span class="p">,</span> <span class="nx">middleware</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">group</span><span class="p">.</span><span class="nx">returnObj</span><span class="p">()</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">group</span> <span class="o">*</span><span class="nx">RouterGroup</span><span class="p">)</span> <span class="nx">combineHandlers</span><span class="p">(</span><span class="nx">handlers</span> <span class="nx">HandlersChain</span><span class="p">)</span> <span class="nx">HandlersChain</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">finalSize</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">group</span><span class="p">.</span><span class="nx">Handlers</span><span class="p">)</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="nx">handlers</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">finalSize</span> <span class="o">&gt;=</span> <span class="nb">int</span><span class="p">(</span><span class="nx">abortIndex</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;too many handlers&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="nx">mergedHandlers</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="nx">HandlersChain</span><span class="p">,</span> <span class="nx">finalSize</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">copy</span><span class="p">(</span><span class="nx">mergedHandlers</span><span class="p">,</span> <span class="nx">group</span><span class="p">.</span><span class="nx">Handlers</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">copy</span><span class="p">(</span><span class="nx">mergedHandlers</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="nx">group</span><span class="p">.</span><span class="nx">Handlers</span><span class="p">):],</span> <span class="nx">handlers</span><span class="p">)</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">mergedHandlers</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span></code></pre></td></tr></table></div></figure>


<p>可以看到它内嵌了一个结构体<code>RouterGroup</code>，那么它是干嘛的呢？</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// RouterGroup is used internally to configure router, a RouterGroup is associated with</span>
</span><span class='line'><span class="c1">// a prefix and an array of handlers (middleware).</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">RouterGroup</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">Handlers</span> <span class="nx">HandlersChain</span>
</span><span class='line'>    <span class="nx">basePath</span> <span class="kt">string</span>
</span><span class='line'>    <span class="nx">engine</span>   <span class="o">*</span><span class="nx">Engine</span>
</span><span class='line'>    <span class="nx">root</span>     <span class="kt">bool</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>从注释上看，这就是负责路由映射的了。上面示例代码中的<code>r.GET</code>实际调用的是<code>RouterGroup</code>的方法(thanks to struct embedding)</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// GET is a shortcut for router.Handle(&quot;GET&quot;, path, handle).</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">group</span> <span class="o">*</span><span class="nx">RouterGroup</span><span class="p">)</span> <span class="nx">GET</span><span class="p">(</span><span class="nx">relativePath</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">handlers</span> <span class="o">...</span><span class="nx">HandlerFunc</span><span class="p">)</span> <span class="nx">IRoutes</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">group</span><span class="p">.</span><span class="nx">handle</span><span class="p">(</span><span class="s">&quot;GET&quot;</span><span class="p">,</span> <span class="nx">relativePath</span><span class="p">,</span> <span class="nx">handlers</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">group</span> <span class="o">*</span><span class="nx">RouterGroup</span><span class="p">)</span> <span class="nx">handle</span><span class="p">(</span><span class="nx">httpMethod</span><span class="p">,</span> <span class="nx">relativePath</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">handlers</span> <span class="nx">HandlersChain</span><span class="p">)</span> <span class="nx">IRoutes</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">absolutePath</span> <span class="o">:=</span> <span class="nx">group</span><span class="p">.</span><span class="nx">calculateAbsolutePath</span><span class="p">(</span><span class="nx">relativePath</span><span class="p">)</span>
</span><span class='line'>    <span class="c1">// 如果在group上定义了共用的handler，比如logger和recovery此处会和传入的handlers一同返回</span>
</span><span class='line'>    <span class="nx">handlers</span> <span class="p">=</span> <span class="nx">group</span><span class="p">.</span><span class="nx">combineHandlers</span><span class="p">(</span><span class="nx">handlers</span><span class="p">)</span>
</span><span class='line'>    <span class="c1">// 这里应该是建立请求路径和handler的映射关系</span>
</span><span class='line'>    <span class="c1">// 涉及到Redix tree算法，感兴趣的可以具体看看</span>
</span><span class='line'>    <span class="nx">group</span><span class="p">.</span><span class="nx">engine</span><span class="p">.</span><span class="nx">addRoute</span><span class="p">(</span><span class="nx">httpMethod</span><span class="p">,</span> <span class="nx">absolutePath</span><span class="p">,</span> <span class="nx">handlers</span><span class="p">)</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">group</span><span class="p">.</span><span class="nx">returnObj</span><span class="p">()</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>再看一个复杂一点的例子：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>  <span class="c1">// Creates a router without any middleware by default</span>
</span><span class='line'>  <span class="nx">r</span> <span class="o">:=</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">New</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Global middleware</span>
</span><span class='line'>  <span class="c1">// Logger middleware will write the logs to gin.DefaultWriter even if you set with GIN_MODE=release.</span>
</span><span class='line'>  <span class="c1">// By default gin.DefaultWriter = os.Stdout</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">Use</span><span class="p">(</span><span class="nx">gin</span><span class="p">.</span><span class="nx">Logger</span><span class="p">())</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Recovery middleware recovers from any panics and writes a 500 if there was one.</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">Use</span><span class="p">(</span><span class="nx">gin</span><span class="p">.</span><span class="nx">Recovery</span><span class="p">())</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Per route middleware, you can add as many as you desire.</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">GET</span><span class="p">(</span><span class="s">&quot;/benchmark&quot;</span><span class="p">,</span> <span class="nx">MyBenchLogger</span><span class="p">(),</span> <span class="nx">benchEndpoint</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Authorization group</span>
</span><span class='line'>  <span class="c1">// authorized := r.Group(&quot;/&quot;, AuthRequired())</span>
</span><span class='line'>  <span class="c1">// exactly the same as:</span>
</span><span class='line'>  <span class="nx">authorized</span> <span class="o">:=</span> <span class="nx">r</span><span class="p">.</span><span class="nx">Group</span><span class="p">(</span><span class="s">&quot;/&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="c1">// per group middleware! in this case we use the custom created</span>
</span><span class='line'>  <span class="c1">// AuthRequired() middleware just in the &quot;authorized&quot; group.</span>
</span><span class='line'>  <span class="nx">authorized</span><span class="p">.</span><span class="nx">Use</span><span class="p">(</span><span class="nx">AuthRequired</span><span class="p">())</span>
</span><span class='line'>  <span class="p">{</span>
</span><span class='line'>      <span class="nx">authorized</span><span class="p">.</span><span class="nx">POST</span><span class="p">(</span><span class="s">&quot;/login&quot;</span><span class="p">,</span> <span class="nx">loginEndpoint</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">authorized</span><span class="p">.</span><span class="nx">POST</span><span class="p">(</span><span class="s">&quot;/submit&quot;</span><span class="p">,</span> <span class="nx">submitEndpoint</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">authorized</span><span class="p">.</span><span class="nx">POST</span><span class="p">(</span><span class="s">&quot;/read&quot;</span><span class="p">,</span> <span class="nx">readEndpoint</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// nested group</span>
</span><span class='line'>      <span class="nx">testing</span> <span class="o">:=</span> <span class="nx">authorized</span><span class="p">.</span><span class="nx">Group</span><span class="p">(</span><span class="s">&quot;testing&quot;</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">testing</span><span class="p">.</span><span class="nx">GET</span><span class="p">(</span><span class="s">&quot;/analytics&quot;</span><span class="p">,</span> <span class="nx">analyticsEndpoint</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Listen and serve on 0.0.0.0:8080</span>
</span><span class='line'>  <span class="nx">r</span><span class="p">.</span><span class="nx">Run</span><span class="p">(</span><span class="s">&quot;:8080&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>从上面的例子中，我们知道还可以定义组的概念（对应url中有统一前缀的情况），而且还可以针对这个组来绑定统一的middleware，那么是怎么实现的呢？</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// Group creates a new router group. You should add all the routes that have common middlewares or the same path prefix.</span>
</span><span class='line'><span class="c1">// For example, all the routes that use a common middleware for authorization could be grouped.</span>
</span><span class='line'><span class="c1">// 返回的依然是一个RouterGroup，但，是一个新的RouterGroup对象</span>
</span><span class='line'><span class="c1">// 且，Handlers和basePath都基于base group做了相应的更新</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">group</span> <span class="o">*</span><span class="nx">RouterGroup</span><span class="p">)</span> <span class="nx">Group</span><span class="p">(</span><span class="nx">relativePath</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">handlers</span> <span class="o">...</span><span class="nx">HandlerFunc</span><span class="p">)</span> <span class="o">*</span><span class="nx">RouterGroup</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="o">&amp;</span><span class="nx">RouterGroup</span><span class="p">{</span>
</span><span class='line'>        <span class="nx">Handlers</span><span class="p">:</span> <span class="nx">group</span><span class="p">.</span><span class="nx">combineHandlers</span><span class="p">(</span><span class="nx">handlers</span><span class="p">),</span>
</span><span class='line'>        <span class="nx">basePath</span><span class="p">:</span> <span class="nx">group</span><span class="p">.</span><span class="nx">calculateAbsolutePath</span><span class="p">(</span><span class="nx">relativePath</span><span class="p">),</span>
</span><span class='line'>        <span class="nx">engine</span><span class="p">:</span>   <span class="nx">group</span><span class="p">.</span><span class="nx">engine</span><span class="p">,</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>以上就做好了url与处理逻辑的绑定，下面再看一下是如何Run起来的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Run attaches the router to a http.Server and starts listening and serving HTTP requests.</span>
</span><span class='line'>  <span class="c1">// It is a shortcut for http.ListenAndServe(addr, router)</span>
</span><span class='line'>  <span class="c1">// Note: this method will block the calling goroutine indefinitely unless an error happens.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span><span class="p">)</span> <span class="nx">Run</span><span class="p">(</span><span class="nx">addr</span> <span class="o">...</span><span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">defer</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span> <span class="nx">debugPrintError</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">}()</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">address</span> <span class="o">:=</span> <span class="nx">resolveAddress</span><span class="p">(</span><span class="nx">addr</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">debugPrint</span><span class="p">(</span><span class="s">&quot;Listening and serving HTTP on %s\n&quot;</span><span class="p">,</span> <span class="nx">address</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">err</span> <span class="p">=</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">address</span><span class="p">,</span> <span class="nx">engine</span><span class="p">)</span>
</span><span class='line'>      <span class="k">return</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>从上面看，就是直接调用了http包的ListenAndServe，嗯，那engine一定是实现了http要求的接口了</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// ServeHTTP conforms to the http.Handler interface.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span><span class="p">)</span> <span class="nx">ServeHTTP</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">req</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 从池子中取出一个ctx</span>
</span><span class='line'>      <span class="nx">c</span> <span class="o">:=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">pool</span><span class="p">.</span><span class="nx">Get</span><span class="p">().(</span><span class="o">*</span><span class="nx">Context</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">writermem</span><span class="p">.</span><span class="nx">reset</span><span class="p">(</span><span class="nx">w</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 请求放在了ctx的Request字段上</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span> <span class="p">=</span> <span class="nx">req</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">reset</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">handleHTTPRequest</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">pool</span><span class="p">.</span><span class="nx">Put</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>可以看到，只要实现这一个方法就能无缝对接http包的Server功能，完全不用care服务端的listen、accept、connect这些细节，只关注具体的处理逻辑就行了，这太值了！</p>

<p>从<code>ServeHTTP</code>的代码看出，这里的逻辑也很简略，就是从对象池取出一个Context，然后把它扔给<code>handleHTTPRequest</code>而已。关于Context的实现我们待会再细说，现在先重点分析下<code>handleHTTPRequest</code>相关的逻辑，上面设置了那么多middleware和handler，那么具体是怎么被执行的呢？</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span><span class="p">)</span> <span class="nx">handleHTTPRequest</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 解析出method、path</span>
</span><span class='line'>      <span class="nx">httpMethod</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">Method</span>
</span><span class='line'>      <span class="nx">rPath</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span>
</span><span class='line'>      <span class="nx">unescape</span> <span class="o">:=</span> <span class="kc">false</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">UseRawPath</span> <span class="o">&amp;&amp;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">RawPath</span><span class="p">)</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rPath</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">RawPath</span>
</span><span class='line'>          <span class="nx">unescape</span> <span class="p">=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">UnescapePathValues</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">rPath</span> <span class="p">=</span> <span class="nx">cleanPath</span><span class="p">(</span><span class="nx">rPath</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Find root of the tree for the given HTTP method</span>
</span><span class='line'>      <span class="c1">// 然后开始在路由树中匹配</span>
</span><span class='line'>      <span class="nx">t</span> <span class="o">:=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">trees</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">tl</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="nx">t</span><span class="p">);</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">tl</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">t</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">method</span> <span class="o">!=</span> <span class="nx">httpMethod</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">continue</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">root</span> <span class="o">:=</span> <span class="nx">t</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">root</span>
</span><span class='line'>          <span class="c1">// Find route in tree</span>
</span><span class='line'>          <span class="c1">// 如果能匹配上，则会返回这个路径对应的handlers</span>
</span><span class='line'>          <span class="nx">handlers</span><span class="p">,</span> <span class="nx">params</span><span class="p">,</span> <span class="nx">tsr</span> <span class="o">:=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">getValue</span><span class="p">(</span><span class="nx">rPath</span><span class="p">,</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Params</span><span class="p">,</span> <span class="nx">unescape</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">handlers</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="c1">// 注意全都赋给了Context</span>
</span><span class='line'>              <span class="c1">// 艰巨的任务交给了Context同志</span>
</span><span class='line'>              <span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span> <span class="p">=</span> <span class="nx">handlers</span>
</span><span class='line'>              <span class="nx">c</span><span class="p">.</span><span class="nx">Params</span> <span class="p">=</span> <span class="nx">params</span>
</span><span class='line'>              <span class="c1">// 关键点，只是调用了一下Next()！！</span>
</span><span class='line'>              <span class="nx">c</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span>
</span><span class='line'>              <span class="nx">c</span><span class="p">.</span><span class="nx">writermem</span><span class="p">.</span><span class="nx">WriteHeaderNow</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">httpMethod</span> <span class="o">!=</span> <span class="s">&quot;CONNECT&quot;</span> <span class="o">&amp;&amp;</span> <span class="nx">rPath</span> <span class="o">!=</span> <span class="s">&quot;/&quot;</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">tsr</span> <span class="o">&amp;&amp;</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">RedirectTrailingSlash</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">redirectTrailingSlash</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>                  <span class="k">return</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">RedirectFixedPath</span> <span class="o">&amp;&amp;</span> <span class="nx">redirectFixedPath</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">root</span><span class="p">,</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">RedirectFixedPath</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">return</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">break</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 路径匹配但方法不匹配的情况</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">HandleMethodNotAllowed</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">tree</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">trees</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">tree</span><span class="p">.</span><span class="nx">method</span> <span class="o">==</span> <span class="nx">httpMethod</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">continue</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">handlers</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">tree</span><span class="p">.</span><span class="nx">root</span><span class="p">.</span><span class="nx">getValue</span><span class="p">(</span><span class="nx">rPath</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">unescape</span><span class="p">);</span> <span class="nx">handlers</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span> <span class="p">=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">allNoMethod</span>
</span><span class='line'>                  <span class="nx">serveError</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">http</span><span class="p">.</span><span class="nx">StatusMethodNotAllowed</span><span class="p">,</span> <span class="nx">default405Body</span><span class="p">)</span>
</span><span class='line'>                  <span class="k">return</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 没找到路径的情况</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span> <span class="p">=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">allNoRoute</span>
</span><span class='line'>      <span class="nx">serveError</span><span class="p">(</span><span class="nx">c</span><span class="p">,</span> <span class="nx">http</span><span class="p">.</span><span class="nx">StatusNotFound</span><span class="p">,</span> <span class="nx">default404Body</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么问题来了，<code>c.Next()</code>是何方神圣？尽管说要等下再分析Context，但，已经等不及了..</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Next should be used only inside middleware.</span>
</span><span class='line'>  <span class="c1">// It executes the pending handlers in the chain inside the calling handler.</span>
</span><span class='line'>  <span class="c1">// See example in GitHub.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">Next</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">index</span><span class="o">++</span>
</span><span class='line'>      <span class="c1">// 注意c.index初始化时是被置为-1的，因此</span>
</span><span class='line'>      <span class="c1">// 逻辑也就是把c.handlers从头遍历、执行一遍</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">c</span><span class="p">.</span><span class="nx">index</span> <span class="p">&lt;</span> <span class="nb">int8</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span><span class="p">))</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span><span class="p">[</span><span class="nx">c</span><span class="p">.</span><span class="nx">index</span><span class="p">](</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">index</span><span class="o">++</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>擦，竟然丧心病狂的只有这么短？而且逻辑也很清楚，就是把handlers依次执行一遍，这能实现普通的handler的功能我信，但能实现middleware的功能我是不信的，顺便科普下大名鼎鼎的middleware的基本功能：</p>

<blockquote><p>I use the term middleware, but each language/framework calls the concept differently. NodeJS and Rails calls it middleware. In the Java enterprise world (i.e. Java Servlet), it’s called filters. C# calls it delegate handlers. <strong>Essentially, the middleware performs some specific function on the HTTP request or response at a specific stage in the HTTP pipeline before or after the user defined controller.</strong> Middleware is a design pattern to eloquently add cross cutting concerns like logging, handling authentication, or gzip compression without having many code contact points. refer: <a href="https://www.moesif.com/blog/engineering/middleware/What-Is-HTTP-Middleware/#">What is HTTP middleware?</a></p></blockquote>

<p>一个http中间件基本上会分别在用户自定义业务逻辑之前和之后执行，所以一个含有中间件的业务的执行逻辑应该是这样：</p>

<ul>
<li>中间件的pre_request逻辑</li>
<li>用户自定义业务逻辑</li>
<li>中间件的post_request逻辑</li>
</ul>


<p>可以看到中间件的逻辑是被用户自定义逻辑给割裂开了，分两段执行，用一张图来表示可能更清楚：</p>

<p><img src="https://i.imgur.com/96NE89V.png" alt="" /></p>

<p>直接找一个内置middleware的实现来分析下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// LoggerWithConfig instance a Logger middleware with config.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">LoggerWithConfig</span><span class="p">(</span><span class="nx">conf</span> <span class="nx">LoggerConfig</span><span class="p">)</span> <span class="nx">HandlerFunc</span> <span class="p">{</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// ...</span>
</span><span class='line'>        <span class="c1">// 前面省略一堆配置logger的逻辑</span>
</span><span class='line'>        <span class="c1">// ...</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="c1">// Start timer</span>
</span><span class='line'>        <span class="nx">start</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">()</span>
</span><span class='line'>        <span class="nx">path</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span>
</span><span class='line'>        <span class="nx">raw</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">RawQuery</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// Process request</span>
</span><span class='line'>        <span class="c1">// 这里也调了c.Next()！！</span>
</span><span class='line'>        <span class="nx">c</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// Log only when path is not being skipped</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">skip</span><span class="p">[</span><span class="nx">path</span><span class="p">];</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">param</span> <span class="o">:=</span> <span class="nx">LogFormatterParams</span><span class="p">{</span>
</span><span class='line'>                <span class="nx">Request</span><span class="p">:</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">,</span>
</span><span class='line'>                <span class="nx">isTerm</span><span class="p">:</span>  <span class="nx">isTerm</span><span class="p">,</span>
</span><span class='line'>                <span class="nx">Keys</span><span class="p">:</span>    <span class="nx">c</span><span class="p">.</span><span class="nx">Keys</span><span class="p">,</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="c1">// Stop timer</span>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">TimeStamp</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">Latency</span> <span class="p">=</span> <span class="nx">param</span><span class="p">.</span><span class="nx">TimeStamp</span><span class="p">.</span><span class="nx">Sub</span><span class="p">(</span><span class="nx">start</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">ClientIP</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">ClientIP</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">Method</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">.</span><span class="nx">Method</span>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">StatusCode</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span><span class="p">.</span><span class="nx">Status</span><span class="p">()</span>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">ErrorMessage</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Errors</span><span class="p">.</span><span class="nx">ByType</span><span class="p">(</span><span class="nx">ErrorTypePrivate</span><span class="p">).</span><span class="nx">String</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">BodySize</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span><span class="p">.</span><span class="nx">Size</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>            <span class="k">if</span> <span class="nx">raw</span> <span class="o">!=</span> <span class="s">&quot;&quot;</span> <span class="p">{</span>
</span><span class='line'>                <span class="nx">path</span> <span class="p">=</span> <span class="nx">path</span> <span class="o">+</span> <span class="s">&quot;?&quot;</span> <span class="o">+</span> <span class="nx">raw</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">param</span><span class="p">.</span><span class="nx">Path</span> <span class="p">=</span> <span class="nx">path</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">out</span><span class="p">,</span> <span class="nx">formatter</span><span class="p">(</span><span class="nx">param</span><span class="p">))</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>可以看到，玄机在于middleware里也调了一次<code>c.Next()</code>，那为什么这样就能实现middleware的功能呢？我们来分析一下：</p>

<p>假设除了用户业务逻辑外，只有这一个middleware，那么，执行流程会是这样：</p>

<ul>
<li><code>handleHTTPRequest</code> 中的<code>c.Next()</code>触发了Logger middleware的执行</li>
<li>Logger middleware先执行了直到<code>c.Next()</code>之前的逻辑</li>
<li>又进入<code>c.Next()</code>函数，此时index++，开始下一个handler函数，即用户自定义业务逻辑</li>
<li>执行用户自定义业务逻辑完成后返回</li>
<li>此时，第二步中的<code>c.Next()</code>调用执行完成</li>
<li>开始执行<code>c.Next()</code>后边的逻辑</li>
<li>完成</li>
</ul>


<p>可以看到，里面涉及到了对<code>c.Next()</code>的递归调用，而middleware功能的实现正是借助对函数的递归调用和层层返回来实现的。</p>

<p>middleware的实现逻辑理清楚后我们来正式看下Context的实现，这个号称是Gin里最重要的结构：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Context is the most important part of gin. It allows us to pass variables between middleware,</span>
</span><span class='line'>  <span class="c1">// manage the flow, validate the JSON of a request and render a JSON response for example.</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">Context</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">writermem</span> <span class="nx">responseWriter</span>
</span><span class='line'>      <span class="nx">Request</span>   <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span>
</span><span class='line'>      <span class="nx">Writer</span>    <span class="nx">ResponseWriter</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">Params</span>   <span class="nx">Params</span>
</span><span class='line'>      <span class="nx">handlers</span> <span class="nx">HandlersChain</span>
</span><span class='line'>      <span class="nx">index</span>    <span class="kt">int8</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Keys is a key/value pair exclusively for the context of each request.</span>
</span><span class='line'>      <span class="nx">Keys</span> <span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kd">interface</span><span class="p">{}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Errors is a list of errors attached to all the handlers/middlewares who used this context.</span>
</span><span class='line'>      <span class="nx">Errors</span> <span class="nx">errorMsgs</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Accepted defines a list of manually accepted formats for content negotiation.</span>
</span><span class='line'>      <span class="nx">Accepted</span> <span class="p">[]</span><span class="kt">string</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>通过注释我们看到，它的作用有：流程控制（即实现middleware）、校验和绑定request和渲染response。流程控制我们前面已经分析过了，下面重点分析下后两个是怎么做的</p>

<p>先分析request的处理，先来个例子看下怎么用的：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// Binding from JSON</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">Login</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">User</span>     <span class="kt">string</span> <span class="s">`form:&quot;user&quot; json:&quot;user&quot; xml:&quot;user&quot;  binding:&quot;required&quot;`</span>
</span><span class='line'>  <span class="nx">Password</span> <span class="kt">string</span> <span class="s">`form:&quot;password&quot; json:&quot;password&quot; xml:&quot;password&quot; binding:&quot;required&quot;`</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">router</span> <span class="o">:=</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">Default</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Example for binding JSON ({&quot;user&quot;: &quot;manu&quot;, &quot;password&quot;: &quot;123&quot;})</span>
</span><span class='line'>  <span class="nx">router</span><span class="p">.</span><span class="nx">POST</span><span class="p">(</span><span class="s">&quot;/loginJSON&quot;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">gin</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">json</span> <span class="nx">Login</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">ShouldBindJSON</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">json</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">JSON</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">,</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">H</span><span class="p">{</span><span class="s">&quot;error&quot;</span><span class="p">:</span> <span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">()})</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      
</span><span class='line'>      <span class="k">if</span> <span class="nx">json</span><span class="p">.</span><span class="nx">User</span> <span class="o">!=</span> <span class="s">&quot;manu&quot;</span> <span class="o">||</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Password</span> <span class="o">!=</span> <span class="s">&quot;123&quot;</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">JSON</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusUnauthorized</span><span class="p">,</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">H</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;unauthorized&quot;</span><span class="p">})</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">JSON</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusOK</span><span class="p">,</span> <span class="nx">gin</span><span class="p">.</span><span class="nx">H</span><span class="p">{</span><span class="s">&quot;status&quot;</span><span class="p">:</span> <span class="s">&quot;you are logged in&quot;</span><span class="p">})</span>
</span><span class='line'>  <span class="p">})</span>
</span><span class='line'>  
</span><span class='line'>    <span class="c1">// Listen and serve on 0.0.0.0:8080</span>
</span><span class='line'>  <span class="nx">router</span><span class="p">.</span><span class="nx">Run</span><span class="p">(</span><span class="s">&quot;:8080&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>可以看到它通过一个<code>c.ShouldBindJSON</code>可以把request data（json格式的）绑定到自定义的struct上，它还支持很多其它方法来绑定其它格式的输入，比如：</p>

<ul>
<li>ShouldBindXML</li>
<li>ShouldBindQuery</li>
<li>ShouldBindYAML</li>
<li>ShouldBindHeader</li>
</ul>


<p>我们来具体分析下它的实现原理，拿一个<code>BindJSON</code>来看，其它的都类似</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// BindJSON is a shortcut for c.MustBindWith(obj, binding.JSON).</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">BindJSON</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">c</span><span class="p">.</span><span class="nx">MustBindWith</span><span class="p">(</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">binding</span><span class="p">.</span><span class="nx">JSON</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// MustBindWith binds the passed struct pointer using the specified binding engine.</span>
</span><span class='line'>  <span class="c1">// It will abort the request with HTTP 400 if any error occurs.</span>
</span><span class='line'>  <span class="c1">// See the binding package.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">MustBindWith</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{},</span> <span class="nx">b</span> <span class="nx">binding</span><span class="p">.</span><span class="nx">Binding</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">ShouldBindWith</span><span class="p">(</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">b</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">AbortWithError</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">,</span> <span class="nx">err</span><span class="p">).</span><span class="nx">SetType</span><span class="p">(</span><span class="nx">ErrorTypeBind</span><span class="p">)</span> <span class="c1">// nolint: errcheck</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// ShouldBindWith binds the passed struct pointer using the specified binding engine.</span>
</span><span class='line'>  <span class="c1">// See the binding package.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">ShouldBindWith</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{},</span> <span class="nx">b</span> <span class="nx">binding</span><span class="p">.</span><span class="nx">Binding</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">b</span><span class="p">.</span><span class="nx">Bind</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">Request</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Binding describes the interface which needs to be implemented for binding the</span>
</span><span class='line'>  <span class="c1">// data present in the request such as JSON request body, query parameters or</span>
</span><span class='line'>  <span class="c1">// the form POST.</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">Binding</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">Name</span><span class="p">()</span> <span class="kt">string</span>
</span><span class='line'>      <span class="nx">Bind</span><span class="p">(</span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">,</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// These implement the Binding interface and can be used to bind the data</span>
</span><span class='line'>  <span class="c1">// present in the request to struct instances.</span>
</span><span class='line'>  <span class="kd">var</span> <span class="p">(</span>
</span><span class='line'>      <span class="nx">JSON</span>          <span class="p">=</span> <span class="nx">jsonBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">XML</span>           <span class="p">=</span> <span class="nx">xmlBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">Form</span>          <span class="p">=</span> <span class="nx">formBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">Query</span>         <span class="p">=</span> <span class="nx">queryBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">FormPost</span>      <span class="p">=</span> <span class="nx">formPostBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">FormMultipart</span> <span class="p">=</span> <span class="nx">formMultipartBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">ProtoBuf</span>      <span class="p">=</span> <span class="nx">protobufBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">MsgPack</span>       <span class="p">=</span> <span class="nx">msgpackBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">YAML</span>          <span class="p">=</span> <span class="nx">yamlBinding</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">Uri</span>           <span class="p">=</span> <span class="nx">uriBinding</span><span class="p">{}</span>
</span><span class='line'>  <span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kd">type</span> <span class="nx">jsonBinding</span> <span class="kd">struct</span><span class="p">{}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">jsonBinding</span><span class="p">)</span> <span class="nx">Name</span><span class="p">()</span> <span class="kt">string</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="s">&quot;json&quot;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">jsonBinding</span><span class="p">)</span> <span class="nx">Bind</span><span class="p">(</span><span class="nx">req</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">req</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">||</span> <span class="nx">req</span><span class="p">.</span><span class="nx">Body</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">&quot;invalid request&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">decodeJSON</span><span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">Body</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">decodeJSON</span><span class="p">(</span><span class="nx">r</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">decoder</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">EnableDecoderUseNumber</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">decoder</span><span class="p">.</span><span class="nx">UseNumber</span><span class="p">()</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">decoder</span><span class="p">.</span><span class="nx">Decode</span><span class="p">(</span><span class="nx">obj</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">validate</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>可以看到，<code>BindXXX</code> &mdash;> <code>MustBindWith</code> &mdash;> <code>ShouldBindWith</code> &mdash;> <code>b.Bind</code>，所有支持的格式都是一样的模式，后续如果想增加一个新的格式，只需要按照<code>Binding</code>
这个接口来实现一下那两个方法就可以用了。</p>

<p>对参数的校验用到了<a href="https://github.com/go-playground/validator">validator</a>这个包，下面具体分析一下</p>

<p>以 bindJSON 为例，最终会调用到<code>decodeJSON</code>：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">decodeJSON</span><span class="p">(</span><span class="nx">r</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">decoder</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">EnableDecoderUseNumber</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">decoder</span><span class="p">.</span><span class="nx">UseNumber</span><span class="p">()</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">EnableDecoderDisallowUnknownFields</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">decoder</span><span class="p">.</span><span class="nx">DisallowUnknownFields</span><span class="p">()</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">decoder</span><span class="p">.</span><span class="nx">Decode</span><span class="p">(</span><span class="nx">obj</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">validate</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="nx">Validator</span> <span class="nx">StructValidator</span> <span class="p">=</span> <span class="o">&amp;</span><span class="nx">defaultValidator</span><span class="p">{}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// validate的定义在此</span>
</span><span class='line'><span class="c1">// 位于binding/binding.go</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">validate</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">Validator</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">Validator</span><span class="p">.</span><span class="nx">ValidateStruct</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 位于binding/default_validator.go</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">v</span> <span class="o">*</span><span class="nx">defaultValidator</span><span class="p">)</span> <span class="nx">ValidateStruct</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">value</span> <span class="o">:=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">valueType</span> <span class="o">:=</span> <span class="nx">value</span><span class="p">.</span><span class="nx">Kind</span><span class="p">()</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">valueType</span> <span class="o">==</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Ptr</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">valueType</span> <span class="p">=</span> <span class="nx">value</span><span class="p">.</span><span class="nx">Elem</span><span class="p">().</span><span class="nx">Kind</span><span class="p">()</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">valueType</span> <span class="o">==</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Struct</span> <span class="p">{</span>
</span><span class='line'>     <span class="c1">// 懒加载</span>
</span><span class='line'>      <span class="nx">v</span><span class="p">.</span><span class="nx">lazyinit</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">v</span><span class="p">.</span><span class="nx">validate</span><span class="p">.</span><span class="nx">Struct</span><span class="p">(</span><span class="nx">obj</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">v</span> <span class="o">*</span><span class="nx">defaultValidator</span><span class="p">)</span> <span class="nx">lazyinit</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">v</span><span class="p">.</span><span class="nx">once</span><span class="p">.</span><span class="nx">Do</span><span class="p">(</span><span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">v</span><span class="p">.</span><span class="nx">validate</span> <span class="p">=</span> <span class="nx">validator</span><span class="p">.</span><span class="nx">New</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">v</span><span class="p">.</span><span class="nx">validate</span><span class="p">.</span><span class="nx">SetTagName</span><span class="p">(</span><span class="s">&quot;binding&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="p">})</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>下面分析下处理完的结果是如何返回http response的（包括http status、header、resp body等）：</p>

<p>我们知道<code>net/http</code>包中经典函数签名<code>ServeHTTP(w http.ResponseWriter, req *http.Request)</code>，其中这个w就是用于返回response的，向它写入数据即可，那么Gin中是怎么跟它联系上的呢？</p>

<p>仍然需要再看一遍engine的ServeHTTP这个方法</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// ServeHTTP conforms to the http.Handler interface.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">engine</span> <span class="o">*</span><span class="nx">Engine</span><span class="p">)</span> <span class="nx">ServeHTTP</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">req</span> <span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 从池子中取出一个ctx</span>
</span><span class='line'>      <span class="nx">c</span> <span class="o">:=</span> <span class="nx">engine</span><span class="p">.</span><span class="nx">pool</span><span class="p">.</span><span class="nx">Get</span><span class="p">().(</span><span class="o">*</span><span class="nx">Context</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 此处把w绑定到ctx上了</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">writermem</span><span class="p">.</span><span class="nx">reset</span><span class="p">(</span><span class="nx">w</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 请求放在了ctx的Request字段上</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Request</span> <span class="p">=</span> <span class="nx">req</span>
</span><span class='line'>      <span class="c1">// 设置Writer等, 重置handlers，后面handleHTTPRequest会根据请求路径重新计算</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">reset</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">handleHTTPRequest</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">engine</span><span class="p">.</span><span class="nx">pool</span><span class="p">.</span><span class="nx">Put</span><span class="p">(</span><span class="nx">c</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="c1">// 来自：response_writer.go</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">responseWriter</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span>
</span><span class='line'>    <span class="nx">size</span>   <span class="kt">int</span>
</span><span class='line'>    <span class="nx">status</span> <span class="kt">int</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>  <span class="c1">// 来自：response_writer.go</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">w</span> <span class="o">*</span><span class="nx">responseWriter</span><span class="p">)</span> <span class="nx">reset</span><span class="p">(</span><span class="nx">writer</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// w是在这里被绑定到Gin内部的一个结构体responseWriter上的</span>
</span><span class='line'>    <span class="nx">w</span><span class="p">.</span><span class="nx">ResponseWriter</span> <span class="p">=</span> <span class="nx">writer</span>
</span><span class='line'>    <span class="nx">w</span><span class="p">.</span><span class="nx">size</span> <span class="p">=</span> <span class="nx">noWritten</span>
</span><span class='line'>    <span class="nx">w</span><span class="p">.</span><span class="nx">status</span> <span class="p">=</span> <span class="nx">defaultStatus</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>   <span class="c1">// 来自：context.go</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">reset</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span> <span class="p">=</span> <span class="o">&amp;</span><span class="nx">c</span><span class="p">.</span><span class="nx">writermem</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Params</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Params</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">handlers</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="c1">// 初始是-1，那就可以理解Next()中的index++了</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">index</span> <span class="p">=</span> <span class="o">-</span><span class="mi">1</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Keys</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Errors</span> <span class="p">=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Errors</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Accepted</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>思考一个问题：</p>

<blockquote><p>为何有<code>writermem</code>和<code>Writer</code>两个功能类似的字段？它们两个从功能上看都是用于返回response的</p></blockquote>

<p>使用上看，<code>http.ResponseWriter</code>先被赋给了<code>writermem</code>里的一个字段，然后<code>writermem</code>又被赋给了<code>Writer</code>。那么为啥要这么绕呢？我的理解：<code>Writer</code>是gin自己定义的接口，它扩展了Golang的<code>http.ResponseWriter</code>接口，因此直接把<code>http.ResponseWriter</code>赋给<code>Writer</code>肯定不行，只能曲线救国，这样就可以既能享用<code>http.ResponseWriter</code>的好处又能有自己的扩展功能。（其实，感觉只使用<code>writermem</code>这一个字段也能满足，暂时没想明白增加一个自定义接口的作用）</p>

<p>返回的response的格式也可以是多种多样的，比如json、xml、yaml等，是如何支持多种格式的呢？以json为例：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// JSON serializes the given struct as JSON into the response body.</span>
</span><span class='line'>  <span class="c1">// It also sets the Content-Type as &quot;application/json&quot;.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">JSON</span><span class="p">(</span><span class="nx">code</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Render</span><span class="p">(</span><span class="nx">code</span><span class="p">,</span> <span class="nx">render</span><span class="p">.</span><span class="nx">JSON</span><span class="p">{</span><span class="nx">Data</span><span class="p">:</span> <span class="nx">obj</span><span class="p">})</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Render writes the response headers and calls render.Render to render data.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">Render</span><span class="p">(</span><span class="nx">code</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">r</span> <span class="nx">render</span><span class="p">.</span><span class="nx">Render</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">Status</span><span class="p">(</span><span class="nx">code</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">bodyAllowedForStatus</span><span class="p">(</span><span class="nx">code</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">r</span><span class="p">.</span><span class="nx">WriteContentType</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span><span class="p">.</span><span class="nx">WriteHeaderNow</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">r</span><span class="p">.</span><span class="nx">Render</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">Writer</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Status sets the HTTP response code.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Context</span><span class="p">)</span> <span class="nx">Status</span><span class="p">(</span><span class="nx">code</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">c</span><span class="p">.</span><span class="nx">writermem</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">code</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Render interface is to be implemented by JSON, XML, HTML, YAML and so on.</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">Render</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// Render writes data with custom ContentType.</span>
</span><span class='line'>    <span class="nx">Render</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">)</span> <span class="kt">error</span>
</span><span class='line'>    <span class="c1">// WriteContentType writes custom ContentType.</span>
</span><span class='line'>    <span class="nx">WriteContentType</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// JSON contains the given interface object.</span>
</span><span class='line'>  <span class="c1">// 来源：render/json.go</span>
</span><span class='line'>  <span class="c1">// 这就是上面那个render.JSON</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">JSON</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">Data</span> <span class="kd">interface</span><span class="p">{}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Render (JSON) writes data with custom ContentType.</span>
</span><span class='line'>  <span class="c1">// 来源：render/json.go</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">r</span> <span class="nx">JSON</span><span class="p">)</span> <span class="nx">Render</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">)</span> <span class="p">(</span><span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">WriteJSON</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="nx">r</span><span class="p">.</span><span class="nx">Data</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// WriteJSON marshals the given interface object and writes it with custom ContentType.</span>
</span><span class='line'>  <span class="c1">// 来源：render/json.go</span>
</span><span class='line'>  <span class="kd">func</span> <span class="nx">WriteJSON</span><span class="p">(</span><span class="nx">w</span> <span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">writeContentType</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="nx">jsonContentType</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">jsonBytes</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">w</span><span class="p">.</span><span class="nx">Write</span><span class="p">(</span><span class="nx">jsonBytes</span><span class="p">)</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>跟绑定request是一样的套路，<code>c.XXX</code> &mdash;> <code>c.Render</code> &mdash;> <code>r.Render(c.Writer)</code>，后续如果想加新的格式，遵循Render接口实现那两个方法即可</p>

<h3>其它</h3>

<h4>对json的优化</h4>

<p>内置了两种json实现，一个是标准库的<code>encoding/json</code>，另一个是<code>github.com/json-iterator/go</code>，在编译的时候通过指定tag可以选择使用哪种实现。我们看看这是怎么做到的</p>

<p>定义好内部包<code>internal/json</code>，这个包下有两个文件，json.go和jsoniter.go，分别代表标准库和jsoniter的实现，以jsoniter为例来看一下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// +build jsoniter</span>
</span><span class='line'>
</span><span class='line'><span class="kn">package</span> <span class="nx">json</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="s">&quot;github.com/json-iterator/go&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="p">(</span>
</span><span class='line'>    <span class="nx">json</span> <span class="p">=</span> <span class="nx">jsoniter</span><span class="p">.</span><span class="nx">ConfigCompatibleWithStandardLibrary</span>
</span><span class='line'>    <span class="c1">// Marshal is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">Marshal</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span>
</span><span class='line'>    <span class="c1">// Unmarshal is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">Unmarshal</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span>
</span><span class='line'>    <span class="c1">// MarshalIndent is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">MarshalIndent</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">MarshalIndent</span>
</span><span class='line'>    <span class="c1">// NewDecoder is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">NewDecoder</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span>
</span><span class='line'>    <span class="c1">// NewEncoder is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">NewEncoder</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewEncoder</span>
</span><span class='line'><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>只是import相应的包，然后再把一些公开方法export出来，注意到最开头的一行：<code>// +build jsoniter</code>，它的意思是，当指定编译tag为<code>jsoniter</code>时，使用当前文件来编译</p>

<p>对比另一个文件， 它的开头写着：<code>// +build !jsoniter</code>，表示当tag不是<code>jsoniter</code>时用这个文件编译</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// +build !jsoniter</span>
</span><span class='line'>
</span><span class='line'><span class="kn">package</span> <span class="nx">json</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="s">&quot;encoding/json&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="p">(</span>
</span><span class='line'>    <span class="c1">// Marshal is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">Marshal</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span>
</span><span class='line'>    <span class="c1">// Unmarshal is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">Unmarshal</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span>
</span><span class='line'>    <span class="c1">// MarshalIndent is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">MarshalIndent</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">MarshalIndent</span>
</span><span class='line'>    <span class="c1">// NewDecoder is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">NewDecoder</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewDecoder</span>
</span><span class='line'>    <span class="c1">// NewEncoder is exported by gin/json package.</span>
</span><span class='line'>    <span class="nx">NewEncoder</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">NewEncoder</span>
</span><span class='line'><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后，在使用的地方import内部封装的包：<code>github.com/gin-gonic/gin/internal/json</code></p>

<p>最后，当编译时需指定tag：<code>$ go build -tags=jsoniter .</code></p>

<p><a href="https://www.flysnow.org/2017/11/05/go-auto-choice-json-libs.html">Go语言中自动选择json解析库</a></p>

<h3>参考</h3>

<ul>
<li><a href="https://juejin.im/post/5b38a6b16fb9a00e6714ab3a">拆轮子系列：gin框架</a></li>
<li><a href="http://www.itarea.cn/golang/57.html">Golang gin框架源码解析</a></li>
<li><a href="https://xguox.me/gin-source-code.html/">从 net/http 入门到 Gin 源码梳理</a></li>
<li><a href="https://github.com/hhstore/blog/issues/131">gin 源码剖析</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[database/sql 源码学习]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/10/17/golang-database-sql/"/>
    <updated>2019-10-17T23:30:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/10/17/golang-database-sql</id>
    <content type="html"><![CDATA[<h3>数据结构</h3>

<h4>ColumnType</h4>

<p>contains the name and type of a column. 用于表示数据库表字段的基本信息</p>

<h4>Conn</h4>

<p>represents a single database connection rather than a pool of database connections. 只代表一个数据库连接，适用于只能对一个连接操作的场景，比如开启事务。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">type</span> <span class="nx">Conn</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// closemu prevents the connection from closing while there</span>
</span><span class='line'>      <span class="c1">// is an active query. It is held for read during queries</span>
</span><span class='line'>      <span class="c1">// and exclusively during close.</span>
</span><span class='line'>      <span class="nx">closemu</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">RWMutex</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// dc is owned until close, at which point</span>
</span><span class='line'>      <span class="c1">// it&#39;s returned to the connection pool.</span>
</span><span class='line'>      <span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// done transitions from 0 to 1 exactly once, on close.</span>
</span><span class='line'>      <span class="c1">// Once done, all operations fail with ErrConnDone.</span>
</span><span class='line'>      <span class="c1">// Use atomic operations on value when checking value.</span>
</span><span class='line'>      <span class="nx">done</span> <span class="kt">int32</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h4>DB</h4>

<p>a database handle representing a pool of zero or more underlying connections. It&rsquo;s safe for concurrent use by multiple goroutines. 代表一个数据库连接池</p>

<p>The sql package creates and frees connections automatically; it also maintains a free pool of idle connections. 连接池由包自动管理</p>

<p>Once DB.Begin is called, the returned Tx is bound to a single connection.Once Commit or Rollback is called on the transaction, that transaction&rsquo;s connection is returned to DB&rsquo;s idle connection pool. 如果是事务的话，在事务提交或回滚前始终用的是一个连接</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">type</span> <span class="nx">DB</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// Atomic access only. At top of struct to prevent mis-alignment</span>
</span><span class='line'>      <span class="c1">// on 32-bit platforms. Of type time.Duration.</span>
</span><span class='line'>      <span class="nx">waitDuration</span> <span class="kt">int64</span> <span class="c1">// Total time waited for new connections.</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">connector</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Connector</span>
</span><span class='line'>      <span class="c1">// numClosed is an atomic counter which represents a total number of</span>
</span><span class='line'>      <span class="c1">// closed connections. Stmt.openStmt checks it before cleaning closed</span>
</span><span class='line'>      <span class="c1">// connections in Stmt.css.</span>
</span><span class='line'>      <span class="nx">numClosed</span> <span class="kt">uint64</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">mu</span>           <span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span> <span class="c1">// protects following fields</span>
</span><span class='line'>      <span class="nx">freeConn</span>     <span class="p">[]</span><span class="o">*</span><span class="nx">driverConn</span>
</span><span class='line'>      <span class="nx">connRequests</span> <span class="kd">map</span><span class="p">[</span><span class="kt">uint64</span><span class="p">]</span><span class="kd">chan</span> <span class="nx">connRequest</span>
</span><span class='line'>      <span class="nx">nextRequest</span>  <span class="kt">uint64</span> <span class="c1">// Next key to use in connRequests.</span>
</span><span class='line'>      <span class="nx">numOpen</span>      <span class="kt">int</span>    <span class="c1">// number of opened and pending open connections</span>
</span><span class='line'>      <span class="c1">// Used to signal the need for new connections</span>
</span><span class='line'>      <span class="c1">// a goroutine running connectionOpener() reads on this chan and</span>
</span><span class='line'>      <span class="c1">// maybeOpenNewConnections sends on the chan (one send per needed connection)</span>
</span><span class='line'>      <span class="c1">// It is closed during db.Close(). The close tells the connectionOpener</span>
</span><span class='line'>      <span class="c1">// goroutine to exit.</span>
</span><span class='line'>      <span class="nx">openerCh</span>          <span class="kd">chan</span> <span class="kd">struct</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">resetterCh</span>        <span class="kd">chan</span> <span class="o">*</span><span class="nx">driverConn</span>
</span><span class='line'>      <span class="nx">closed</span>            <span class="kt">bool</span>
</span><span class='line'>      <span class="nx">dep</span>               <span class="kd">map</span><span class="p">[</span><span class="nx">finalCloser</span><span class="p">]</span><span class="nx">depSet</span>
</span><span class='line'>      <span class="nx">lastPut</span>           <span class="kd">map</span><span class="p">[</span><span class="o">*</span><span class="nx">driverConn</span><span class="p">]</span><span class="kt">string</span> <span class="c1">// stacktrace of last conn&#39;s put; debug only</span>
</span><span class='line'>      <span class="nx">maxIdle</span>           <span class="kt">int</span>                    <span class="c1">// zero means defaultMaxIdleConns; negative means 0</span>
</span><span class='line'>      <span class="nx">maxOpen</span>           <span class="kt">int</span>                    <span class="c1">// &lt;= 0 means unlimited</span>
</span><span class='line'>      <span class="nx">maxLifetime</span>       <span class="nx">time</span><span class="p">.</span><span class="nx">Duration</span>          <span class="c1">// maximum amount of time a connection may be reused</span>
</span><span class='line'>      <span class="nx">cleanerCh</span>         <span class="kd">chan</span> <span class="kd">struct</span><span class="p">{}</span>
</span><span class='line'>      <span class="nx">waitCount</span>         <span class="kt">int64</span> <span class="c1">// Total number of connections waited for.</span>
</span><span class='line'>      <span class="nx">maxIdleClosed</span>     <span class="kt">int64</span> <span class="c1">// Total number of connections closed due to idle.</span>
</span><span class='line'>      <span class="nx">maxLifetimeClosed</span> <span class="kt">int64</span> <span class="c1">// Total number of connections closed due to max free limit.</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">stop</span> <span class="kd">func</span><span class="p">()</span> <span class="c1">// stop cancels the connection opener and the session resetter.</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h4>driverConn</h4>

<p>wraps a driver.Conn with a mutex, to be held during all calls into the Conn 封装了 driver.Conn</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">type</span> <span class="nx">driverConn</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">db</span>        <span class="o">*</span><span class="nx">DB</span>
</span><span class='line'>      <span class="nx">createdAt</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span>  <span class="c1">// guards following</span>
</span><span class='line'>      <span class="nx">ci</span>          <span class="nx">driver</span><span class="p">.</span><span class="nx">Conn</span>
</span><span class='line'>      <span class="nx">closed</span>      <span class="kt">bool</span>
</span><span class='line'>      <span class="nx">finalClosed</span> <span class="kt">bool</span> <span class="c1">// ci.Close has been called</span>
</span><span class='line'>      <span class="nx">openStmt</span>    <span class="kd">map</span><span class="p">[</span><span class="o">*</span><span class="nx">driverStmt</span><span class="p">]</span><span class="kt">bool</span>
</span><span class='line'>      <span class="nx">lastErr</span>     <span class="kt">error</span> <span class="c1">// lastError captures the result of the session resetter.</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// guarded by db.mu</span>
</span><span class='line'>      <span class="nx">inUse</span>      <span class="kt">bool</span>
</span><span class='line'>      <span class="nx">onPut</span>      <span class="p">[]</span><span class="kd">func</span><span class="p">()</span> <span class="c1">// code (with db.mu held) run when conn is next returned</span>
</span><span class='line'>      <span class="nx">dbmuClosed</span> <span class="kt">bool</span>     <span class="c1">// same as closed, but guarded by db.mu, for removeClosedStmtLocked</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h4>Rows</h4>

<p>the result of a query. Its cursor starts before the first row of the result set. Use Next to advance from row to row. 代表查询结果集</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">type</span> <span class="nx">Rows</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">dc</span>          <span class="o">*</span><span class="nx">driverConn</span> <span class="c1">// owned; must call releaseConn when closed to release</span>
</span><span class='line'>      <span class="nx">releaseConn</span> <span class="kd">func</span><span class="p">(</span><span class="kt">error</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">rowsi</span>       <span class="nx">driver</span><span class="p">.</span><span class="nx">Rows</span>
</span><span class='line'>      <span class="nx">cancel</span>      <span class="kd">func</span><span class="p">()</span>      <span class="c1">// called when Rows is closed, may be nil.</span>
</span><span class='line'>      <span class="nx">closeStmt</span>   <span class="o">*</span><span class="nx">driverStmt</span> <span class="c1">// if non-nil, statement to Close on close</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// closemu prevents Rows from closing while there</span>
</span><span class='line'>      <span class="c1">// is an active streaming result. It is held for read during non-close operations</span>
</span><span class='line'>      <span class="c1">// and exclusively during close.</span>
</span><span class='line'>      <span class="c1">//</span>
</span><span class='line'>      <span class="c1">// closemu guards lasterr and closed.</span>
</span><span class='line'>      <span class="nx">closemu</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">RWMutex</span>
</span><span class='line'>      <span class="nx">closed</span>  <span class="kt">bool</span>
</span><span class='line'>      <span class="nx">lasterr</span> <span class="kt">error</span> <span class="c1">// non-nil only if closed is true</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// lastcols is only used in Scan, Next, and NextResultSet which are expected</span>
</span><span class='line'>      <span class="c1">// not to be called concurrently.</span>
</span><span class='line'>      <span class="nx">lastcols</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Value</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>总结一下：</p>

<p>DB对象代表了一个连接池，它来维护这个池子的生长</p>

<p>driverConn代表了一个具体的数据库连接，每一条sql的执行最终都会落实到一个具体的driverConn</p>

<p>一次Query查询会返回一个Rows对象，代表的是一个查询集合，里面是一行一行的记录</p>

<p>其它公开的数据结构（Stmt、Tx）都是构建于上面说的结构之上的</p>

<h3>接口</h3>

<p><code>database/sql</code>包里没有对任何一个数据库协议的具体实现，它只是用一堆接口描述了任何一个想要接入该包的数据库驱动需要实现的函数</p>

<h4>driver.Conn</h4>

<p>a connection to a database. It is not used concurrently by multiple goroutines.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">type</span> <span class="nx">Conn</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// Prepare returns a prepared statement, bound to this connection.</span>
</span><span class='line'>      <span class="nx">Prepare</span><span class="p">(</span><span class="nx">query</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="nx">Stmt</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Close invalidates and potentially stops any current</span>
</span><span class='line'>      <span class="c1">// prepared statements and transactions, marking this</span>
</span><span class='line'>      <span class="c1">// connection as no longer in use.</span>
</span><span class='line'>      <span class="c1">//</span>
</span><span class='line'>      <span class="c1">// Because the sql package maintains a free pool of</span>
</span><span class='line'>      <span class="c1">// connections and only calls Close when there&#39;s a surplus of</span>
</span><span class='line'>      <span class="c1">// idle connections, it shouldn&#39;t be necessary for drivers to</span>
</span><span class='line'>      <span class="c1">// do their own connection caching.</span>
</span><span class='line'>      <span class="nx">Close</span><span class="p">()</span> <span class="kt">error</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Begin starts and returns a new transaction.</span>
</span><span class='line'>      <span class="c1">//</span>
</span><span class='line'>      <span class="c1">// Deprecated: Drivers should implement ConnBeginTx instead (or additionally).</span>
</span><span class='line'>      <span class="nx">Begin</span><span class="p">()</span> <span class="p">(</span><span class="nx">Tx</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>注意，Conn 接口要求必须实现Prepare方法</p>

<h3>流程分析</h3>

<p>从官方文档的一个例子开始</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kn">package</span> <span class="nx">main</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="p">(</span>
</span><span class='line'>  <span class="s">&quot;context&quot;</span>
</span><span class='line'>  <span class="s">&quot;database/sql&quot;</span>
</span><span class='line'>  <span class="s">&quot;flag&quot;</span>
</span><span class='line'>  <span class="s">&quot;log&quot;</span>
</span><span class='line'>  <span class="s">&quot;os&quot;</span>
</span><span class='line'>  <span class="s">&quot;os/signal&quot;</span>
</span><span class='line'>  <span class="s">&quot;time&quot;</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="nx">pool</span> <span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span> <span class="c1">// Database connection pool.</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">id</span> <span class="o">:=</span> <span class="nx">flag</span><span class="p">.</span><span class="nx">Int64</span><span class="p">(</span><span class="s">&quot;id&quot;</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s">&quot;person ID to find&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">dsn</span> <span class="o">:=</span> <span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">&quot;dsn&quot;</span><span class="p">,</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Getenv</span><span class="p">(</span><span class="s">&quot;DSN&quot;</span><span class="p">),</span> <span class="s">&quot;connection data source name&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="o">*</span><span class="nx">dsn</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">&quot;missing dsn flag&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">*</span><span class="nx">id</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">&quot;missing person ID&quot;</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Opening a driver typically will not attempt to connect to the database.</span>
</span><span class='line'>  <span class="nx">pool</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">sql</span><span class="p">.</span><span class="nx">Open</span><span class="p">(</span><span class="s">&quot;driver-name&quot;</span><span class="p">,</span> <span class="o">*</span><span class="nx">dsn</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// This will not be a connection error, but a DSN parse error or</span>
</span><span class='line'>      <span class="c1">// another initialization error.</span>
</span><span class='line'>      <span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">&quot;unable to use data source name&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">defer</span> <span class="nx">pool</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">pool</span><span class="p">.</span><span class="nx">SetConnMaxLifetime</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">pool</span><span class="p">.</span><span class="nx">SetMaxIdleConns</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">pool</span><span class="p">.</span><span class="nx">SetMaxOpenConns</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">ctx</span><span class="p">,</span> <span class="nx">stop</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithCancel</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">())</span>
</span><span class='line'>  <span class="k">defer</span> <span class="nx">stop</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">appSignal</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Signal</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
</span><span class='line'>  <span class="nx">signal</span><span class="p">.</span><span class="nx">Notify</span><span class="p">(</span><span class="nx">appSignal</span><span class="p">,</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Interrupt</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">appSignal</span><span class="p">:</span>
</span><span class='line'>          <span class="nx">stop</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}()</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">Ping</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">Query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="o">*</span><span class="nx">id</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Ping the database to verify DSN provided by the user is valid and the</span>
</span><span class='line'><span class="c1">// server accessible. If the ping fails exit the program with an error.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">Ping</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">ctx</span><span class="p">,</span> <span class="nx">cancel</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithTimeout</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
</span><span class='line'>  <span class="k">defer</span> <span class="nx">cancel</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">pool</span><span class="p">.</span><span class="nx">PingContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">log</span><span class="p">.</span><span class="nx">Fatalf</span><span class="p">(</span><span class="s">&quot;unable to connect to database: %v&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Query the database for the information requested and prints the results.</span>
</span><span class='line'><span class="c1">// If the query fails exit the program with an error.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">Query</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">id</span> <span class="kt">int64</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">ctx</span><span class="p">,</span> <span class="nx">cancel</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithTimeout</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">5</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
</span><span class='line'>  <span class="k">defer</span> <span class="nx">cancel</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">name</span> <span class="kt">string</span>
</span><span class='line'>  <span class="nx">err</span> <span class="o">:=</span> <span class="nx">pool</span><span class="p">.</span><span class="nx">QueryRowContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="s">&quot;select p.name from people as p where p.id = :id;&quot;</span><span class="p">,</span> <span class="nx">sql</span><span class="p">.</span><span class="nx">Named</span><span class="p">(</span><span class="s">&quot;id&quot;</span><span class="p">,</span> <span class="nx">id</span><span class="p">)).</span><span class="nx">Scan</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">name</span><span class="p">)</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">&quot;unable to execute search query&quot;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;name=&quot;</span><span class="p">,</span> <span class="nx">name</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>首先用<code>Open</code>生成了一个数据库连接池对象，我们看看它是怎么做的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">var</span> <span class="p">(</span>
</span><span class='line'>       <span class="c1">// 一把读写锁</span>
</span><span class='line'>      <span class="nx">driversMu</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">RWMutex</span>
</span><span class='line'>      <span class="c1">// 全局的驱动映射map</span>
</span><span class='line'>      <span class="nx">drivers</span>   <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Driver</span><span class="p">)</span>
</span><span class='line'>  <span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="nx">Open</span><span class="p">(</span><span class="nx">driverName</span><span class="p">,</span> <span class="nx">dataSourceName</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">DB</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">driversMu</span><span class="p">.</span><span class="nx">RLock</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">driveri</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">drivers</span><span class="p">[</span><span class="nx">driverName</span><span class="p">]</span>
</span><span class='line'>      <span class="nx">driversMu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">&quot;sql: unknown driver %q (forgotten import?)&quot;</span><span class="p">,</span> <span class="nx">driverName</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">driverCtx</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">driveri</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">DriverContext</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">connector</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">driverCtx</span><span class="p">.</span><span class="nx">OpenConnector</span><span class="p">(</span><span class="nx">dataSourceName</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">OpenDB</span><span class="p">(</span><span class="nx">connector</span><span class="p">),</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">OpenDB</span><span class="p">(</span><span class="nx">dsnConnector</span><span class="p">{</span><span class="nx">dsn</span><span class="p">:</span> <span class="nx">dataSourceName</span><span class="p">,</span> <span class="nx">driver</span><span class="p">:</span> <span class="nx">driveri</span><span class="p">}),</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="nx">OpenDB</span><span class="p">(</span><span class="nx">c</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Connector</span><span class="p">)</span> <span class="o">*</span><span class="nx">DB</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">ctx</span><span class="p">,</span> <span class="nx">cancel</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithCancel</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">())</span>
</span><span class='line'>      <span class="nx">db</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">DB</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">connector</span><span class="p">:</span>    <span class="nx">c</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">openerCh</span><span class="p">:</span>     <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{},</span> <span class="nx">connectionRequestQueueSize</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">resetterCh</span><span class="p">:</span>   <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="mi">50</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">lastPut</span><span class="p">:</span>      <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="o">*</span><span class="nx">driverConn</span><span class="p">]</span><span class="kt">string</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">connRequests</span><span class="p">:</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">uint64</span><span class="p">]</span><span class="kd">chan</span> <span class="nx">connRequest</span><span class="p">),</span>
</span><span class='line'>          <span class="nx">stop</span><span class="p">:</span>         <span class="nx">cancel</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 注意这里起了两个goroutine</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// 上面的DB的结构体的openerCh字段的注释里说明了</span>
</span><span class='line'>      <span class="c1">// 这个goroutine的处理逻辑</span>
</span><span class='line'>      <span class="k">go</span> <span class="nx">db</span><span class="p">.</span><span class="nx">connectionOpener</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>      <span class="k">go</span> <span class="nx">db</span><span class="p">.</span><span class="nx">connectionResetter</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么一次查询是如何进行的呢？</p>

<p>DB.Query 最终会调用到 Conn.Query</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// QueryContext executes a query that returns rows, typically a SELECT.</span>
</span><span class='line'>  <span class="c1">// The args are for any placeholder parameters in the query.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">QueryContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">rows</span> <span class="o">*</span><span class="nx">Rows</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">maxBadConnRetries</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">,</span> <span class="nx">cachedOrNewConn</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">break</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">,</span> <span class="nx">alwaysNewConn</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Query executes a query that returns rows, typically a SELECT.</span>
</span><span class='line'>  <span class="c1">// The args are for any placeholder parameters in the query.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">Query</span><span class="p">(</span><span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">QueryContext</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">(),</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{},</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 获取一个driverConn</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">queryDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// QueryContext executes a query that returns rows, typically a SELECT.</span>
</span><span class='line'>  <span class="c1">// The args are for any placeholder parameters in the query.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">Conn</span><span class="p">)</span> <span class="nx">QueryContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">release</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">grabConn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">c</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">queryDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">release</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>先获取一个connection，这个过程我们暂且不看，继续往下追</p>

<p>从queryDC开始进入驱动的代码</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// queryDC executes a query on the given connection.</span>
</span><span class='line'>  <span class="c1">// The connection gets released by the releaseConn function.</span>
</span><span class='line'>  <span class="c1">// The ctx context is from a query method and the txctx context is from an</span>
</span><span class='line'>  <span class="c1">// optional transaction context.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">queryDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">txctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">releaseConn</span> <span class="kd">func</span><span class="p">(</span><span class="kt">error</span><span class="p">),</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 判断驱动的连接对象是否实现了Queryer接口</span>
</span><span class='line'>      <span class="nx">queryerCtx</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">QueryerContext</span><span class="p">)</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">queryer</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Queryer</span>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">queryer</span><span class="p">,</span> <span class="nx">ok</span> <span class="p">=</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Queryer</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">nvdargs</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">NamedValue</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">rowsi</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Rows</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>          <span class="nx">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">nvdargs</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">driverArgsConnLocked</span><span class="p">(</span><span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">return</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="nx">rowsi</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">ctxDriverQuery</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">queryerCtx</span><span class="p">,</span> <span class="nx">queryer</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">nvdargs</span><span class="p">)</span>
</span><span class='line'>          <span class="p">})</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrSkip</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 说明用户没有要求用Prepared Statement</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">releaseConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>                  <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="c1">// Note: ownership of dc passes to the *Rows, to be freed</span>
</span><span class='line'>              <span class="c1">// with releaseConn.</span>
</span><span class='line'>              <span class="c1">// 注意：这里将驱动返回的行信息rowsi在这里封装成了</span>
</span><span class='line'>              <span class="c1">// sql.Rows</span>
</span><span class='line'>              <span class="nx">rows</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">Rows</span><span class="p">{</span>
</span><span class='line'>                  <span class="nx">dc</span><span class="p">:</span>          <span class="nx">dc</span><span class="p">,</span>
</span><span class='line'>                  <span class="nx">releaseConn</span><span class="p">:</span> <span class="nx">releaseConn</span><span class="p">,</span>
</span><span class='line'>                  <span class="nx">rowsi</span><span class="p">:</span>       <span class="nx">rowsi</span><span class="p">,</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="nx">rows</span><span class="p">.</span><span class="nx">initContextClose</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">txctx</span><span class="p">)</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 如果驱动没有实现Queryer接口或者用户指定要使用Prepared Statement(返回了driver.ErrSkip），则使用Prepared Statement来做查询</span>
</span><span class='line'>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">si</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Stmt</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="nx">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// prepare</span>
</span><span class='line'>          <span class="nx">si</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">ctxDriverPrepare</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">,</span> <span class="nx">query</span><span class="p">)</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">releaseConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">ds</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">driverStmt</span><span class="p">{</span><span class="nx">Locker</span><span class="p">:</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">si</span><span class="p">:</span> <span class="nx">si</span><span class="p">}</span>
</span><span class='line'>      <span class="c1">// 执行查询</span>
</span><span class='line'>      <span class="nx">rowsi</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">rowsiFromStatement</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">,</span> <span class="nx">ds</span><span class="p">,</span> <span class="nx">args</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">ds</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">releaseConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Note: ownership of ci passes to the *Rows, to be freed</span>
</span><span class='line'>      <span class="c1">// with releaseConn.</span>
</span><span class='line'>      <span class="nx">rows</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">Rows</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">:</span>          <span class="nx">dc</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">releaseConn</span><span class="p">:</span> <span class="nx">releaseConn</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">rowsi</span><span class="p">:</span>       <span class="nx">rowsi</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">closeStmt</span><span class="p">:</span>   <span class="nx">ds</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">rows</span><span class="p">.</span><span class="nx">initContextClose</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">txctx</span><span class="p">)</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>根据驱动或者用户配置不同，以上代码会有不同的执行路径，假设最终走直接查询的路径（即不走Prepared Statement），则会继续执行如下逻辑：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 执行到ctxDriverQuery函数</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">ctxDriverQuery</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">queryerCtx</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">QueryerContext</span><span class="p">,</span> <span class="nx">queryer</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Queryer</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">nvdargs</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">NamedValue</span><span class="p">)</span> <span class="p">(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">queryerCtx</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">queryerCtx</span><span class="p">.</span><span class="nx">QueryContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">nvdargs</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="nx">dargs</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">namedValueToValue</span><span class="p">(</span><span class="nx">nvdargs</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// 注意这里ctx的用法，根据ctx是否已被取消，提早返回 </span>
</span><span class='line'>    <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">default</span><span class="p">:</span>
</span><span class='line'>    <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Err</span><span class="p">()</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="c1">// 从这里以后就会到驱动的代码了</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">queryer</span><span class="p">.</span><span class="nx">Query</span><span class="p">(</span><span class="nx">query</span><span class="p">,</span> <span class="nx">dargs</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>如果使用的是mysql的驱动，会继续执行以下代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">mc</span> <span class="o">*</span><span class="nx">mysqlConn</span><span class="p">)</span> <span class="nx">Query</span><span class="p">(</span><span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Value</span><span class="p">)</span> <span class="p">(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">mc</span> <span class="o">*</span><span class="nx">mysqlConn</span><span class="p">)</span> <span class="nx">query</span><span class="p">(</span><span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Value</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">textRows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">closed</span><span class="p">.</span><span class="nx">IsSet</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">errLog</span><span class="p">.</span><span class="nx">Print</span><span class="p">(</span><span class="nx">ErrInvalidConn</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">args</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 是否转义sql中的参数来防止sql注入</span>
</span><span class='line'>          <span class="c1">// 如果指定不转义的话会使用 prepared statement</span>
</span><span class='line'>          <span class="k">if</span> <span class="p">!</span><span class="nx">mc</span><span class="p">.</span><span class="nx">cfg</span><span class="p">.</span><span class="nx">InterpolateParams</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrSkip</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// try client-side prepare to reduce roundtrip</span>
</span><span class='line'>          <span class="nx">prepared</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">interpolateParams</span><span class="p">(</span><span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">query</span> <span class="p">=</span> <span class="nx">prepared</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// Send command</span>
</span><span class='line'>      <span class="nx">err</span> <span class="o">:=</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">writeCommandPacketStr</span><span class="p">(</span><span class="nx">comQuery</span><span class="p">,</span> <span class="nx">query</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Read Result</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">resLen</span> <span class="kt">int</span>
</span><span class='line'>          <span class="nx">resLen</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">readResultSetHeaderPacket</span><span class="p">()</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">rows</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="nx">textRows</span><span class="p">)</span>
</span><span class='line'>              <span class="nx">rows</span><span class="p">.</span><span class="nx">mc</span> <span class="p">=</span> <span class="nx">mc</span>
</span><span class='line'>
</span><span class='line'>              <span class="k">if</span> <span class="nx">resLen</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">rows</span><span class="p">.</span><span class="nx">rs</span><span class="p">.</span><span class="nx">done</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>
</span><span class='line'>                  <span class="k">switch</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">NextResultSet</span><span class="p">();</span> <span class="nx">err</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">case</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span><span class="p">:</span>
</span><span class='line'>                      <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>                  <span class="k">default</span><span class="p">:</span>
</span><span class='line'>                      <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>                  <span class="p">}</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>              <span class="c1">// Columns</span>
</span><span class='line'>              <span class="nx">rows</span><span class="p">.</span><span class="nx">rs</span><span class="p">.</span><span class="nx">columns</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">readColumns</span><span class="p">(</span><span class="nx">resLen</span><span class="p">)</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">mc</span><span class="p">.</span><span class="nx">markBadConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>上面只是读出了“表头”，拿到了row对象，下面要开始读取具体的记录了（对应一开始的例子中的Scan语句）</p>

<p>对于只返回一行的query直接调用Scan就可以了(其实，在暗地里帮你调用了Next)，返回多行时需要先调用Next，然后再调用Scan</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">rs</span> <span class="o">*</span><span class="nx">Rows</span><span class="p">)</span> <span class="nx">Next</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">doClose</span><span class="p">,</span> <span class="nx">ok</span> <span class="kt">bool</span>
</span><span class='line'>      <span class="nx">withLock</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RLocker</span><span class="p">(),</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">doClose</span><span class="p">,</span> <span class="nx">ok</span> <span class="p">=</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">nextLocked</span><span class="p">()</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">doClose</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rs</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">ok</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">rs</span> <span class="o">*</span><span class="nx">Rows</span><span class="p">)</span> <span class="nx">nextLocked</span><span class="p">()</span> <span class="p">(</span><span class="nx">doClose</span><span class="p">,</span> <span class="nx">ok</span> <span class="kt">bool</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">false</span><span class="p">,</span> <span class="kc">false</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Lock the driver connection before calling the driver interface</span>
</span><span class='line'>      <span class="c1">// rowsi to prevent a Tx from rolling back the connection at the same time.</span>
</span><span class='line'>      <span class="nx">rs</span><span class="p">.</span><span class="nx">dc</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">defer</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">dc</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span> <span class="p">=</span> <span class="nb">make</span><span class="p">([]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Value</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">rowsi</span><span class="p">.</span><span class="nx">Columns</span><span class="p">()))</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 从此进入驱动的代码</span>
</span><span class='line'>      <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span> <span class="p">=</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">rowsi</span><span class="p">.</span><span class="nx">Next</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Close the connection if there is a driver error.</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span> <span class="o">!=</span> <span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">true</span><span class="p">,</span> <span class="kc">false</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">nextResultSet</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">rowsi</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">RowsNextResultSet</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">true</span><span class="p">,</span> <span class="kc">false</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// The driver is at the end of the current result set.</span>
</span><span class='line'>          <span class="c1">// Test to see if there is another result set after the current one.</span>
</span><span class='line'>          <span class="c1">// Only close Rows if there is no further result sets to read.</span>
</span><span class='line'>          <span class="k">if</span> <span class="p">!</span><span class="nx">nextResultSet</span><span class="p">.</span><span class="nx">HasNextResultSet</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">doClose</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">doClose</span><span class="p">,</span> <span class="kc">false</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">false</span><span class="p">,</span> <span class="kc">true</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// mysql驱动的代码，位于rows.go中</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">rows</span> <span class="o">*</span><span class="nx">textRows</span><span class="p">)</span> <span class="nx">Next</span><span class="p">(</span><span class="nx">dest</span> <span class="p">[]</span><span class="nx">driver</span><span class="p">.</span><span class="nx">Value</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">mc</span> <span class="o">:=</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">mc</span><span class="p">;</span> <span class="nx">mc</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">mc</span><span class="p">.</span><span class="nb">error</span><span class="p">();</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>            <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// Fetch next row from stream</span>
</span><span class='line'>        <span class="c1">// 这里会读取MySQL的查询结果</span>
</span><span class='line'>        <span class="c1">// 想了解MySQL的client与server间的交互协议的可以看这个函数</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">readRow</span><span class="p">(</span><span class="nx">dest</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行Next之后，会将从mysql读到的行数据保存在Rows对象的一个私有字段里(lastcols)，然后终于来到了Scan阶段</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">rs</span> <span class="o">*</span><span class="nx">Rows</span><span class="p">)</span> <span class="nx">Scan</span><span class="p">(</span><span class="nx">dest</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RLock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span> <span class="o">!=</span> <span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterr</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">err</span> <span class="o">:=</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lasterrOrErrLocked</span><span class="p">(</span><span class="nx">errRowsClosed</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">rs</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">errors</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">&quot;sql: Scan called without calling Next&quot;</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">dest</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">&quot;sql: expected %d destination arguments in Scan, not %d&quot;</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="nx">dest</span><span class="p">))</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 遍历这一行的每一个字段</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">sv</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">lastcols</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 转成Golang内部类型</span>
</span><span class='line'>          <span class="c1">// 对于数据库里的某个字段类型会被转成Golang里哪种类型</span>
</span><span class='line'>          <span class="c1">// 有疑问的可以看这个函数的实现</span>
</span><span class='line'>          <span class="nx">err</span> <span class="o">:=</span> <span class="nx">convertAssignRows</span><span class="p">(</span><span class="nx">dest</span><span class="p">[</span><span class="nx">i</span><span class="p">],</span> <span class="nx">sv</span><span class="p">,</span> <span class="nx">rs</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Errorf</span><span class="p">(</span><span class="s">`sql: Scan error on column index %d, name %q: %v`</span><span class="p">,</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">rs</span><span class="p">.</span><span class="nx">rowsi</span><span class="p">.</span><span class="nx">Columns</span><span class="p">()[</span><span class="nx">i</span><span class="p">],</span> <span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>至此，一次完整的查询过程就分析完了。</p>

<p>下面想具体了解下<code>database/sql</code>包是如何维护数据库的连接池的。</p>

<blockquote><p>有兴趣的可以看一下Goang 从1.0到现在的1.12，<code>database/sql</code>包是如何演化的，如果你去看1.1的源码会发现那时候的连接池实现很简陋，那会儿的连接池没有最大连接数的限制</p></blockquote>

<p>这个连接池有几个特征参数：</p>

<ul>
<li>maxLifetime

<ul>
<li>maximum amount of time a connection may be reused</li>
</ul>
</li>
<li>maxIdle

<ul>
<li>允许的最大空闲连接数</li>
</ul>
</li>
<li>maxOpen

<ul>
<li>允许的打开的最大连接数</li>
</ul>
</li>
</ul>


<p>连接池中的连接数最大会增长到maxOpen，当请求高峰过后，连接池的可用连接数又会慢慢缩回到maxIdle</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">Query</span><span class="p">(</span><span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">QueryContext</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">(),</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">QueryContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">rows</span> <span class="o">*</span><span class="nx">Rows</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="c1">// 优先用池子里的连接</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">maxBadConnRetries</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">,</span> <span class="nx">cachedOrNewConn</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">break</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 当然池子里的连接有可能会过期，如果重试两次还取不到可以用的连接</span>
</span><span class='line'>      <span class="c1">// 那就从数据库请求一个新的</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">,</span> <span class="nx">alwaysNewConn</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">query</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{},</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">queryDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>下面就是获取连接的逻辑了，流程如下：</p>

<ul>
<li>如果策略是优先使用缓存连接且连接池中还有空闲连接，则直接从连接池中取一个连接返回</li>
<li>如果连接池里当前已打开的连接数已超出maxOpen限制，则阻塞一直等待有连接归还到连接池后再取用</li>
<li>否则就新建一个连接返回</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
<span class='line-number'>84</span>
<span class='line-number'>85</span>
<span class='line-number'>86</span>
<span class='line-number'>87</span>
<span class='line-number'>88</span>
<span class='line-number'>89</span>
<span class='line-number'>90</span>
<span class='line-number'>91</span>
<span class='line-number'>92</span>
<span class='line-number'>93</span>
<span class='line-number'>94</span>
<span class='line-number'>95</span>
<span class='line-number'>96</span>
<span class='line-number'>97</span>
<span class='line-number'>98</span>
<span class='line-number'>99</span>
<span class='line-number'>100</span>
<span class='line-number'>101</span>
<span class='line-number'>102</span>
<span class='line-number'>103</span>
<span class='line-number'>104</span>
<span class='line-number'>105</span>
<span class='line-number'>106</span>
<span class='line-number'>107</span>
<span class='line-number'>108</span>
<span class='line-number'>109</span>
<span class='line-number'>110</span>
<span class='line-number'>111</span>
<span class='line-number'>112</span>
<span class='line-number'>113</span>
<span class='line-number'>114</span>
<span class='line-number'>115</span>
<span class='line-number'>116</span>
<span class='line-number'>117</span>
<span class='line-number'>118</span>
<span class='line-number'>119</span>
<span class='line-number'>120</span>
<span class='line-number'>121</span>
<span class='line-number'>122</span>
<span class='line-number'>123</span>
<span class='line-number'>124</span>
<span class='line-number'>125</span>
<span class='line-number'>126</span>
<span class='line-number'>127</span>
<span class='line-number'>128</span>
<span class='line-number'>129</span>
<span class='line-number'>130</span>
<span class='line-number'>131</span>
<span class='line-number'>132</span>
<span class='line-number'>133</span>
<span class='line-number'>134</span>
<span class='line-number'>135</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// conn returns a newly-opened or cached *driverConn.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">errDBClosed</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 检查是否已被取消</span>
</span><span class='line'>      <span class="c1">// Check if the context is expired.</span>
</span><span class='line'>      <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">default</span><span class="p">:</span>
</span><span class='line'>      <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Err</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 一个连接的最大生命周期</span>
</span><span class='line'>      <span class="nx">lifetime</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxLifetime</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Prefer a free connection, if possible.</span>
</span><span class='line'>      <span class="nx">numFree</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">strategy</span> <span class="o">==</span> <span class="nx">cachedOrNewConn</span> <span class="o">&amp;&amp;</span> <span class="nx">numFree</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">conn</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'>          <span class="c1">// copy(dst, src []Type) int</span>
</span><span class='line'>          <span class="c1">// The copy built-in function copies elements from a source slice into a destination slice</span>
</span><span class='line'>          <span class="c1">// The source and destination may overlap.</span>
</span><span class='line'>          <span class="c1">// 这是在更新空闲的数据库连接数组，最开始的那个连接现在已经要被占用了</span>
</span><span class='line'>          <span class="nb">copy</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">,</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</span><span class='line'>          <span class="c1">// 移除掉末尾那个无用的元素</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[:</span><span class="nx">numFree</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
</span><span class='line'>          <span class="c1">// 通过以上两步，成功从freeConn中删掉了开头的第一个连接</span>
</span><span class='line'>          <span class="nx">conn</span><span class="p">.</span><span class="nx">inUse</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="c1">// 检查连接是否已经过期</span>
</span><span class='line'>          <span class="c1">// 注意如果过期并没有尝试取下一个，而是直接返回了错误</span>
</span><span class='line'>          <span class="c1">// 由上层来继续发起连接，我理解这样会保持不同层之间逻辑的干净</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">conn</span><span class="p">.</span><span class="nx">expired</span><span class="p">(</span><span class="nx">lifetime</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">conn</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// Lock around reading lastErr to ensure the session resetter finished.</span>
</span><span class='line'>          <span class="c1">// 这里应该是看下这个连接是不是被session resetter给置为badConn了</span>
</span><span class='line'>          <span class="c1">// 如果是，那也不能用了</span>
</span><span class='line'>          <span class="c1">// session resetter 是一个单独的goroutine在检查每个连接的状态</span>
</span><span class='line'>          <span class="nx">conn</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">err</span> <span class="o">:=</span> <span class="nx">conn</span><span class="p">.</span><span class="nx">lastErr</span>
</span><span class='line'>                    <span class="nx">conn</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">conn</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">conn</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Out of free connections or we were asked not to use one. If we&#39;re not</span>
</span><span class='line'>      <span class="c1">// allowed to open any more connections, make a request and wait.</span>
</span><span class='line'>      <span class="c1">// 如果当前打开的连接数已经超过了db.maxOpen</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxOpen</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span> <span class="o">&gt;=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxOpen</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Make the connRequest channel. It&#39;s buffered so that the</span>
</span><span class='line'>          <span class="c1">// connectionOpener doesn&#39;t block while waiting for the req to be read.</span>
</span><span class='line'>          <span class="nx">req</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="nx">connRequest</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">reqKey</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">nextRequestKeyLocked</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">connRequests</span><span class="p">[</span><span class="nx">reqKey</span><span class="p">]</span> <span class="p">=</span> <span class="nx">req</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">waitCount</span><span class="o">++</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">waitStart</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>          <span class="c1">// Timeout the connection request with the context.</span>
</span><span class='line'>          <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 注意这个select没有default分支，所以会一直阻塞在这两个分支上</span>
</span><span class='line'>          <span class="c1">// 直到任何一个分支的条件满足，即：1. query被取消了 2. 有free连接可用了</span>
</span><span class='line'>          <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>              <span class="c1">// Remove the connection request and ensure no value has been sent</span>
</span><span class='line'>              <span class="c1">// on it after removing.</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>              <span class="nb">delete</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">connRequests</span><span class="p">,</span> <span class="nx">reqKey</span><span class="p">)</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>              <span class="nx">atomic</span><span class="p">.</span><span class="nx">AddInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">db</span><span class="p">.</span><span class="nx">waitDuration</span><span class="p">,</span> <span class="nb">int64</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Since</span><span class="p">(</span><span class="nx">waitStart</span><span class="p">)))</span>
</span><span class='line'>
</span><span class='line'>              <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">default</span><span class="p">:</span>
</span><span class='line'>              <span class="k">case</span> <span class="nx">ret</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">req</span><span class="p">:</span>
</span><span class='line'>                  <span class="k">if</span> <span class="nx">ok</span> <span class="o">&amp;&amp;</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                      <span class="nx">db</span><span class="p">.</span><span class="nx">putConn</span><span class="p">(</span><span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">,</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">err</span><span class="p">,</span> <span class="kc">false</span><span class="p">)</span>
</span><span class='line'>                  <span class="p">}</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Err</span><span class="p">()</span>
</span><span class='line'>          <span class="k">case</span> <span class="nx">ret</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">req</span><span class="p">:</span>
</span><span class='line'>              <span class="nx">atomic</span><span class="p">.</span><span class="nx">AddInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">db</span><span class="p">.</span><span class="nx">waitDuration</span><span class="p">,</span> <span class="nb">int64</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Since</span><span class="p">(</span><span class="nx">waitStart</span><span class="p">)))</span>
</span><span class='line'>
</span><span class='line'>              <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">errDBClosed</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">expired</span><span class="p">(</span><span class="nx">lifetime</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>                  <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">err</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="c1">// Lock around reading lastErr to ensure the session resetter finished.</span>
</span><span class='line'>              <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>              <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">lastErr</span>
</span><span class='line'>              <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>                  <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">conn</span><span class="p">,</span> <span class="nx">ret</span><span class="p">.</span><span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 如果没有空闲连接且尚未达到最大连接数的限制</span>
</span><span class='line'>      <span class="c1">// 那就开一个新连接</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span><span class="o">++</span> <span class="c1">// optimistically</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">ci</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">connector</span><span class="p">.</span><span class="nx">Connect</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span><span class="o">--</span> <span class="c1">// correct for earlier optimism</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">maybeOpenNewConnections</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">dc</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">driverConn</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">:</span>        <span class="nx">db</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">createdAt</span><span class="p">:</span> <span class="nx">nowFunc</span><span class="p">(),</span>
</span><span class='line'>          <span class="nx">ci</span><span class="p">:</span>        <span class="nx">ci</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">inUse</span><span class="p">:</span>     <span class="kc">true</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">addDepLocked</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">dc</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>有借就有还，再借才不难，下面是归还连接的逻辑：</p>

<ul>
<li>如果有connRequests，那么优先满足它</li>
<li>否则就要准备归还到池子中了，但这里还有一个check，如果当前空闲连接书freeConn大于允许的最大空闲连接限制的话就不归还了，会把连接直接关闭掉</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
<span class='line-number'>84</span>
<span class='line-number'>85</span>
<span class='line-number'>86</span>
<span class='line-number'>87</span>
<span class='line-number'>88</span>
<span class='line-number'>89</span>
<span class='line-number'>90</span>
<span class='line-number'>91</span>
<span class='line-number'>92</span>
<span class='line-number'>93</span>
<span class='line-number'>94</span>
<span class='line-number'>95</span>
<span class='line-number'>96</span>
<span class='line-number'>97</span>
<span class='line-number'>98</span>
<span class='line-number'>99</span>
<span class='line-number'>100</span>
<span class='line-number'>101</span>
<span class='line-number'>102</span>
<span class='line-number'>103</span>
<span class='line-number'>104</span>
<span class='line-number'>105</span>
<span class='line-number'>106</span>
<span class='line-number'>107</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// putConn adds a connection to the db&#39;s free pool.</span>
</span><span class='line'>  <span class="c1">// err is optionally the last error that occurred on this connection.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">putConn</span><span class="p">(</span><span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">,</span> <span class="nx">resetSession</span> <span class="kt">bool</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">dc</span><span class="p">.</span><span class="nx">inUse</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">debugGetPut</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;putConn(%v) DUPLICATE was: %s\n\nPREVIOUS was: %s&quot;</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">stack</span><span class="p">(),</span> <span class="nx">db</span><span class="p">.</span><span class="nx">lastPut</span><span class="p">[</span><span class="nx">dc</span><span class="p">])</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;sql: connection returned that was never out&quot;</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">debugGetPut</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">lastPut</span><span class="p">[</span><span class="nx">dc</span><span class="p">]</span> <span class="p">=</span> <span class="nx">stack</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">.</span><span class="nx">inUse</span> <span class="p">=</span> <span class="kc">false</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">fn</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">onPut</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">fn</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">.</span><span class="nx">onPut</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Don&#39;t reuse bad connections.</span>
</span><span class='line'>          <span class="c1">// Since the conn is considered bad and is being discarded, treat it</span>
</span><span class='line'>          <span class="c1">// as closed. Don&#39;t decrement the open count here, finalClose will</span>
</span><span class='line'>          <span class="c1">// take care of that.</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">maybeOpenNewConnections</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">putConnHook</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">putConnHook</span><span class="p">(</span><span class="nx">db</span><span class="p">,</span> <span class="nx">dc</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// Connections do not need to be reset if they will be closed.</span>
</span><span class='line'>          <span class="c1">// Prevents writing to resetterCh after the DB has closed.</span>
</span><span class='line'>          <span class="nx">resetSession</span> <span class="p">=</span> <span class="kc">false</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">resetSession</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">resetSession</span> <span class="p">=</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">.(</span><span class="nx">driver</span><span class="p">.</span><span class="nx">SessionResetter</span><span class="p">);</span> <span class="nx">resetSession</span> <span class="p">{</span>
</span><span class='line'>              <span class="c1">// Lock the driverConn here so it isn&#39;t released until</span>
</span><span class='line'>              <span class="c1">// the connection is reset.</span>
</span><span class='line'>              <span class="c1">// The lock must be taken before the connection is put into</span>
</span><span class='line'>              <span class="c1">// the pool to prevent it from being taken out before it is reset.</span>
</span><span class='line'>              <span class="nx">dc</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">added</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">putConnDBLocked</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kc">nil</span><span class="p">)</span>
</span><span class='line'>      <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">added</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">resetSession</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">dc</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="p">!</span><span class="nx">resetSession</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">default</span><span class="p">:</span>
</span><span class='line'>          <span class="c1">// If the resetterCh is blocking then mark the connection</span>
</span><span class='line'>          <span class="c1">// as bad and continue on.</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">.</span><span class="nx">lastErr</span> <span class="p">=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">case</span> <span class="nx">db</span><span class="p">.</span><span class="nx">resetterCh</span> <span class="o">&lt;-</span> <span class="nx">dc</span><span class="p">:</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">putConnDBLocked</span><span class="p">(</span><span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxOpen</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span> <span class="p">&gt;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxOpen</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 如果有connRequests（连接需求），优先满足它</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">c</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">connRequests</span><span class="p">);</span> <span class="nx">c</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">req</span> <span class="kd">chan</span> <span class="nx">connRequest</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">reqKey</span> <span class="kt">uint64</span>
</span><span class='line'>          <span class="k">for</span> <span class="nx">reqKey</span><span class="p">,</span> <span class="nx">req</span> <span class="p">=</span> <span class="k">range</span> <span class="nx">db</span><span class="p">.</span><span class="nx">connRequests</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">break</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nb">delete</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">connRequests</span><span class="p">,</span> <span class="nx">reqKey</span><span class="p">)</span> <span class="c1">// Remove from pending requests.</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">dc</span><span class="p">.</span><span class="nx">inUse</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">req</span> <span class="o">&lt;-</span> <span class="nx">connRequest</span><span class="p">{</span>
</span><span class='line'>              <span class="nx">conn</span><span class="p">:</span> <span class="nx">dc</span><span class="p">,</span>
</span><span class='line'>              <span class="nx">err</span><span class="p">:</span>  <span class="nx">err</span><span class="p">,</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">true</span>
</span><span class='line'>      <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="p">!</span><span class="nx">db</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxIdleConnsLocked</span><span class="p">()</span> <span class="p">&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>              <span class="c1">// 将连接放回池子</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">,</span> <span class="nx">dc</span><span class="p">)</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">startCleanerLocked</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">true</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// 为啥直接增加了maxIdleClosed而没有做什么呢？</span>
</span><span class='line'>          <span class="c1">// 因为这种情况下本函数会返回false，进而导致</span>
</span><span class='line'>          <span class="c1">// 连接被关闭</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">maxIdleClosed</span><span class="o">++</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>以下逻辑实现连接的生命周期的管理，基本逻辑为：</p>

<p>定期遍历连接池中的每一个连接，检查其是存活时间是否超出了设定的maxLifetime，若超出则将它从池子中删除并关闭它</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// startCleanerLocked starts connectionCleaner if needed.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">startCleanerLocked</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 尽管每次putConnDBLocked的时候都会被调用，但注意它的条件db.cleanerCh == nil</span>
</span><span class='line'>      <span class="c1">// 这意味着其实db.connectionCleaner只会被启动一次</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxLifetime</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">db</span><span class="p">.</span><span class="nx">cleanerCh</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">cleanerCh</span> <span class="p">=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{},</span> <span class="mi">1</span><span class="p">)</span>
</span><span class='line'>          <span class="k">go</span> <span class="nx">db</span><span class="p">.</span><span class="nx">connectionCleaner</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">maxLifetime</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>   <span class="c1">// 清理过期连接的逻辑</span>
</span><span class='line'>   <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">connectionCleaner</span><span class="p">(</span><span class="nx">d</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Duration</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">const</span> <span class="nx">minInterval</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span>
</span><span class='line'>      <span class="c1">// 精度最高为秒级</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">d</span> <span class="p">&lt;</span> <span class="nx">minInterval</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">d</span> <span class="p">=</span> <span class="nx">minInterval</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">t</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">NewTimer</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 这个写法很有意思</span>
</span><span class='line'>          <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">t</span><span class="p">.</span><span class="nx">C</span><span class="p">:</span>
</span><span class='line'>          <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">db</span><span class="p">.</span><span class="nx">cleanerCh</span><span class="p">:</span> <span class="c1">// maxLifetime was changed or db was closed.</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">d</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">maxLifetime</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">db</span><span class="p">.</span><span class="nx">closed</span> <span class="o">||</span> <span class="nx">db</span><span class="p">.</span><span class="nx">numOpen</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">||</span> <span class="nx">d</span> <span class="o">&lt;=</span> <span class="mi">0</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">cleanerCh</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>              <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="nx">expiredSince</span> <span class="o">:=</span> <span class="nx">nowFunc</span><span class="p">().</span><span class="nx">Add</span><span class="p">(</span><span class="o">-</span><span class="nx">d</span><span class="p">)</span>
</span><span class='line'>          <span class="kd">var</span> <span class="nx">closing</span> <span class="p">[]</span><span class="o">*</span><span class="nx">driverConn</span>
</span><span class='line'>          <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">);</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">c</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">c</span><span class="p">.</span><span class="nx">createdAt</span><span class="p">.</span><span class="nx">Before</span><span class="p">(</span><span class="nx">expiredSince</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                  <span class="nx">closing</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">closing</span><span class="p">,</span> <span class="nx">c</span><span class="p">)</span>
</span><span class='line'>                  <span class="nx">last</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span>
</span><span class='line'>                  <span class="c1">// 数组中经典的删除元素的方法</span>
</span><span class='line'>                  <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="nx">last</span><span class="p">]</span>
</span><span class='line'>                  <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[</span><span class="nx">last</span><span class="p">]</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>                  <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">freeConn</span><span class="p">[:</span><span class="nx">last</span><span class="p">]</span>
</span><span class='line'>                  <span class="c1">// 后退一步是为了可以检查刚刚置换过来的元素</span>
</span><span class='line'>                  <span class="nx">i</span><span class="o">--</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">maxLifetimeClosed</span> <span class="o">+=</span> <span class="nb">int64</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">closing</span><span class="p">))</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>          <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">c</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">closing</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">c</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>          <span class="k">if</span> <span class="nx">d</span> <span class="p">&lt;</span> <span class="nx">minInterval</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">d</span> <span class="p">=</span> <span class="nx">minInterval</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="nx">t</span><span class="p">.</span><span class="nx">Reset</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>Prepared Statement</h3>

<p>根据MySQL<a href="https://dev.mysql.com/doc/refman/5.7/en/sql-syntax-prepared-statements.html">文档</a>，prepared statement 只能绑定在一个连接（session）上，即如果在一个连接上执行了prepare后，后续的exec还是要在同一个连接上才行，以下引自MySQL文档：</p>

<blockquote><p>A prepared statement is specific to the session in which it was created. If you terminate a session without deallocating a previously prepared statement, the server deallocates it automatically.</p></blockquote>

<p>但在Golang的<code>database/sql</code>包中，连接并没有直接暴露给开发者（从1.9以后暴露了一个单独的连接对象Conn），开发者看到的是一个连接池，那么<code>database/sql</code>是如何在只暴露连接池的情况下来支持prepared statement的呢？</p>

<p>是这么做的：</p>

<ol>
<li>Prepare的阶段从连接池中取出一个连接来发起prepare的操作，执行完后就把连接放回连接池了，但此时要记住是在这个连接上发起了Prepare</li>
<li>在execute阶段，会从连接池中需要当时发起prepare的那个连接，找到后再继续发起execute操作即可。但这里就有个问题，这时候可能找不到当时那个连接了（要么这个连接正在被别人占用，要么这个连接已经被关闭了），那么此时就需要从池子中再取一个连接出来，重新发起prepare。

<ul>
<li>这里有个坑：在高并发的情况下，会在服务端创建大量重复的prepared statement，可能会耗尽服务端的最大prepared statement限制</li>
</ul>
</li>
</ol>


<p>参考：<a href="http://go-database-sql.org/prepared.html">Using Prepared Statements</a></p>

<p>下面从代码上具体分析一下：</p>

<p>首先是数据结构：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Stmt is a prepared statement.</span>
</span><span class='line'>  <span class="c1">// A Stmt is safe for concurrent use by multiple goroutines.</span>
</span><span class='line'>  <span class="c1">//</span>
</span><span class='line'>  <span class="c1">// If a Stmt is prepared on a Tx or Conn, it will be bound to a single</span>
</span><span class='line'>  <span class="c1">// underlying connection forever. If the Tx or Conn closes, the Stmt will</span>
</span><span class='line'>  <span class="c1">// become unusable and all operations will return an error.</span>
</span><span class='line'>  <span class="c1">// If a Stmt is prepared on a DB, it will remain usable for the lifetime of the</span>
</span><span class='line'>  <span class="c1">// DB. When the Stmt needs to execute on a new underlying connection, it will</span>
</span><span class='line'>  <span class="c1">// prepare itself on the new connection automatically.</span>
</span><span class='line'>  <span class="kd">type</span> <span class="nx">Stmt</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// Immutable:</span>
</span><span class='line'>      <span class="nx">db</span>        <span class="o">*</span><span class="nx">DB</span>    <span class="c1">// where we came from</span>
</span><span class='line'>      <span class="nx">query</span>     <span class="kt">string</span> <span class="c1">// that created the Stmt</span>
</span><span class='line'>      <span class="nx">stickyErr</span> <span class="kt">error</span>  <span class="c1">// if non-nil, this error is returned for all operations</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">closemu</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">RWMutex</span> <span class="c1">// held exclusively during close, for read otherwise.</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// If Stmt is prepared on a Tx or Conn then cg is present and will</span>
</span><span class='line'>      <span class="c1">// only ever grab a connection from cg.</span>
</span><span class='line'>      <span class="c1">// If cg is nil then the Stmt must grab an arbitrary connection</span>
</span><span class='line'>      <span class="c1">// from db and determine if it must prepare the stmt again by</span>
</span><span class='line'>      <span class="c1">// inspecting css.</span>
</span><span class='line'>      <span class="nx">cg</span>   <span class="nx">stmtConnGrabber</span>
</span><span class='line'>      <span class="nx">cgds</span> <span class="o">*</span><span class="nx">driverStmt</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// parentStmt is set when a transaction-specific statement</span>
</span><span class='line'>      <span class="c1">// is requested from an identical statement prepared on the same</span>
</span><span class='line'>      <span class="c1">// conn. parentStmt is used to track the dependency of this statement</span>
</span><span class='line'>      <span class="c1">// on its originating (&quot;parent&quot;) statement so that parentStmt may</span>
</span><span class='line'>      <span class="c1">// be closed by the user without them having to know whether or not</span>
</span><span class='line'>      <span class="c1">// any transactions are still using it.</span>
</span><span class='line'>      <span class="nx">parentStmt</span> <span class="o">*</span><span class="nx">Stmt</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">mu</span>     <span class="nx">sync</span><span class="p">.</span><span class="nx">Mutex</span> <span class="c1">// protects the rest of the fields</span>
</span><span class='line'>      <span class="nx">closed</span> <span class="kt">bool</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// css is a list of underlying driver statement interfaces</span>
</span><span class='line'>      <span class="c1">// that are valid on particular connections. This is only</span>
</span><span class='line'>      <span class="c1">// used if cg == nil and one is found that has idle</span>
</span><span class='line'>      <span class="c1">// connections. If cg != nil, cgds is always used.</span>
</span><span class='line'>      <span class="nx">css</span> <span class="p">[]</span><span class="nx">connStmt</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// lastNumClosed is copied from db.numClosed when Stmt is created</span>
</span><span class='line'>      <span class="c1">// without tx and closed connections in css are removed.</span>
</span><span class='line'>      <span class="nx">lastNumClosed</span> <span class="kt">uint64</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>prepare阶段，分别可以从DB、Conn和Tx上发起Prepare，先分析在DB上的Prepare</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">PrepareContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Stmt</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">stmt</span> <span class="o">*</span><span class="nx">Stmt</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">maxBadConnRetries</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">stmt</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">prepare</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">cachedOrNewConn</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">break</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">prepare</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">alwaysNewConn</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">stmt</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">prepare</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Stmt</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// TODO: check if db.driver supports an optional</span>
</span><span class='line'>      <span class="c1">// driver.Preparer interface and call that instead, if so,</span>
</span><span class='line'>      <span class="c1">// otherwise we make a prepared statement that&#39;s bound</span>
</span><span class='line'>      <span class="c1">// to a connection, and to execute this prepared statement</span>
</span><span class='line'>      <span class="c1">// we either need to use this connection (if it&#39;s free), else</span>
</span><span class='line'>      <span class="c1">// get a new connection + re-prepare + execute on that one.</span>
</span><span class='line'>      <span class="c1">// 先从池子中获取一个连接</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">prepareDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">query</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">prepareDC</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">release</span> <span class="kd">func</span><span class="p">(</span><span class="kt">error</span><span class="p">),</span> <span class="nx">cg</span> <span class="nx">stmtConnGrabber</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Stmt</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">ds</span> <span class="o">*</span><span class="nx">driverStmt</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="k">defer</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">release</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}()</span>
</span><span class='line'>      <span class="c1">// 这里是调用驱动来向server发起Prepare命令</span>
</span><span class='line'>      <span class="nx">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">ds</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">prepareLocked</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">cg</span><span class="p">,</span> <span class="nx">query</span><span class="p">)</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">stmt</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">Stmt</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">:</span>    <span class="nx">db</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">query</span><span class="p">:</span> <span class="nx">query</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">cg</span><span class="p">:</span>    <span class="nx">cg</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">cgds</span><span class="p">:</span>  <span class="nx">ds</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// When cg == nil this statement will need to keep track of various</span>
</span><span class='line'>      <span class="c1">// connections they are prepared on and record the stmt dependency on</span>
</span><span class='line'>      <span class="c1">// the DB.</span>
</span><span class='line'>      <span class="c1">// 记住这个stmt是在哪个conn上Prepare的，便于后续exec时再继续用</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">cg</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">stmt</span><span class="p">.</span><span class="nx">css</span> <span class="p">=</span> <span class="p">[]</span><span class="nx">connStmt</span><span class="p"></span>
</span><span class='line'>          <span class="nx">stmt</span><span class="p">.</span><span class="nx">lastNumClosed</span> <span class="p">=</span> <span class="nx">atomic</span><span class="p">.</span><span class="nx">LoadUint64</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">db</span><span class="p">.</span><span class="nx">numClosed</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">.</span><span class="nx">addDep</span><span class="p">(</span><span class="nx">stmt</span><span class="p">,</span> <span class="nx">stmt</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">stmt</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行阶段</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">Stmt</span><span class="p">)</span> <span class="nx">ExecContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="nx">Result</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RLock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">defer</span> <span class="nx">s</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">res</span> <span class="nx">Result</span>
</span><span class='line'>      <span class="nx">strategy</span> <span class="o">:=</span> <span class="nx">cachedOrNewConn</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">maxBadConnRetries</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">i</span> <span class="o">==</span> <span class="nx">maxBadConnRetries</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">strategy</span> <span class="p">=</span> <span class="nx">alwaysNewConn</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// 这里是尝试取回当时进行Prepare的连接以待继续操作</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">,</span> <span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">ds</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">connStmt</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>                  <span class="k">continue</span>
</span><span class='line'>              <span class="p">}</span>
</span><span class='line'>              <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="c1">// 从驱动读取执行结果</span>
</span><span class='line'>          <span class="nx">res</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">resultFromStatement</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">,</span> <span class="nx">ds</span><span class="p">,</span> <span class="nx">args</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'>          <span class="nx">releaseConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">res</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">Stmt</span><span class="p">)</span> <span class="nx">connStmt</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">releaseConn</span> <span class="kd">func</span><span class="p">(</span><span class="kt">error</span><span class="p">),</span> <span class="nx">ds</span> <span class="o">*</span><span class="nx">driverStmt</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">stickyErr</span><span class="p">;</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">err</span> <span class="p">=</span> <span class="nx">errors</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">&quot;sql: statement is closed&quot;</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// In a transaction or connection, we always use the connection that the</span>
</span><span class='line'>      <span class="c1">// stmt was created on.</span>
</span><span class='line'>      <span class="c1">// 在Tx或者Conn上Prepare的情况</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">cg</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">,</span> <span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">cg</span><span class="p">.</span><span class="nx">grabConn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span> <span class="c1">// blocks, waiting for the connection.</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">return</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">s</span><span class="p">.</span><span class="nx">cgds</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">removeClosedStmtLocked</span><span class="p">()</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>      <span class="c1">// 以下是在DB上Prepare时的处理逻辑</span>
</span><span class='line'>      <span class="c1">// 简单粗暴，直接从池子里取一个连接，</span>
</span><span class='line'>      <span class="c1">// 然后判断这个连接是不是之前prepare过，</span>
</span><span class='line'>      <span class="c1">// 若prepare过，则直接返回这个连接即可，</span>
</span><span class='line'>      <span class="c1">// 若没有，则直接在这个连接上重新prepare后再返回</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">s</span><span class="p">.</span><span class="nx">css</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">v</span><span class="p">.</span><span class="nx">dc</span> <span class="o">==</span> <span class="nx">dc</span> <span class="p">{</span>
</span><span class='line'>              <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>              <span class="k">return</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">v</span><span class="p">.</span><span class="nx">ds</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// No luck; we need to prepare the statement on this connection</span>
</span><span class='line'>      <span class="nx">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">ds</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">prepareOnConnLocked</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">)</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">return</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">ds</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>这里有个小插曲，在对照源码的各个版本的<code>database/sql</code>包的实现时，偶然发现connStmt方法的实现经历了不少的改变呢，想知道这改变背后的原因，于是对提交历史考据了一番，还真找到了：</p>

<p>相关的commit: 1b61a97811626d4c7a8332c107f1e091253d1b2e</p>

<p>对应的github issue: <a href="https://github.com/golang/go/issues/9484">https://github.com/golang/go/issues/9484</a></p>

<p>一开始的实现，相对精细，导致锁冲突有点严重，后来又改成简单粗暴了，同时也减少了锁冲突，提高了性能，但高并发时会有很多重新Prepare的情况。</p>

<p>关闭阶段</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="c1">// Close closes the statement.</span>
</span><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">Stmt</span><span class="p">)</span> <span class="nx">Close</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">defer</span> <span class="nx">s</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">stickyErr</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">s</span><span class="p">.</span><span class="nx">stickyErr</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">closed</span> <span class="p">=</span> <span class="kc">true</span>
</span><span class='line'>      <span class="nx">txds</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">cgds</span>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">cgds</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>
</span><span class='line'>      <span class="nx">s</span><span class="p">.</span><span class="nx">mu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">cg</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">s</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">removeDep</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">s</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">parentStmt</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// If parentStmt is set, we must not close s.txds since it&#39;s stored</span>
</span><span class='line'>          <span class="c1">// in the css array of the parentStmt.</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">s</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">removeDep</span><span class="p">(</span><span class="nx">s</span><span class="p">.</span><span class="nx">parentStmt</span><span class="p">,</span> <span class="nx">s</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// txds是驱动端的stmt，最终调了驱动端的Close来关闭</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">txds</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>参考：</p>

<ul>
<li><a href="https://www.jianshu.com/p/ee0d2e7bef54">Prepared剖析</a></li>
<li><a href="https://dev.mysql.com/doc/refman/5.7/en/sql-syntax-prepared-statements.html">Prepared SQL Statement </a></li>
<li><a href="http://go-database-sql.org/prepared.html">Using Prepared Statements</a></li>
</ul>


<h3>Tx</h3>

<p>下面分析下对数据库事务的支持。同prepared statement一样，在数据库层面事务也只能在一个连接上进行，但在底层实现上，Tx的实现与prepared statement有明显不同，它底层从始至终只使用了一个连接。不过，我不太明白prepared statement为啥没有这样实现。</p>

<p>开启事务</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">BeginTx</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">opts</span> <span class="o">*</span><span class="nx">TxOptions</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">Tx</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">tx</span> <span class="o">*</span><span class="nx">Tx</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">err</span> <span class="kt">error</span>
</span><span class='line'>      <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">maxBadConnRetries</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">tx</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">begin</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">opts</span><span class="p">,</span> <span class="nx">cachedOrNewConn</span><span class="p">)</span>
</span><span class='line'>          <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>              <span class="k">break</span>
</span><span class='line'>          <span class="p">}</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">ErrBadConn</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">begin</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">opts</span><span class="p">,</span> <span class="nx">alwaysNewConn</span><span class="p">)</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">tx</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">begin</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">opts</span> <span class="o">*</span><span class="nx">TxOptions</span><span class="p">,</span> <span class="nx">strategy</span> <span class="nx">connReuseStrategy</span><span class="p">)</span> <span class="p">(</span><span class="nx">tx</span> <span class="o">*</span><span class="nx">Tx</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 依然是先从池子中取出一个空闲连接</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">conn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">strategy</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">db</span><span class="p">.</span><span class="nx">beginDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">releaseConn</span><span class="p">,</span> <span class="nx">opts</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">db</span> <span class="o">*</span><span class="nx">DB</span><span class="p">)</span> <span class="nx">beginDC</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">dc</span> <span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">release</span> <span class="kd">func</span><span class="p">(</span><span class="kt">error</span><span class="p">),</span> <span class="nx">opts</span> <span class="o">*</span><span class="nx">TxOptions</span><span class="p">)</span> <span class="p">(</span><span class="nx">tx</span> <span class="o">*</span><span class="nx">Tx</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="kd">var</span> <span class="nx">txi</span> <span class="nx">driver</span><span class="p">.</span><span class="nx">Tx</span>
</span><span class='line'>      <span class="nx">withLock</span><span class="p">(</span><span class="nx">dc</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="c1">// 调用driver来开启事务</span>
</span><span class='line'>          <span class="nx">txi</span><span class="p">,</span> <span class="nx">err</span> <span class="p">=</span> <span class="nx">ctxDriverBegin</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">opts</span><span class="p">,</span> <span class="nx">dc</span><span class="p">.</span><span class="nx">ci</span><span class="p">)</span>
</span><span class='line'>      <span class="p">})</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">release</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// Schedule the transaction to rollback when the context is cancelled.</span>
</span><span class='line'>      <span class="c1">// The cancel function in Tx will be called after done is set to true.</span>
</span><span class='line'>      <span class="nx">ctx</span><span class="p">,</span> <span class="nx">cancel</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithCancel</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>      <span class="c1">// 注意返回的tx对象记了启动事务的连接dc</span>
</span><span class='line'>      <span class="c1">// 后续所有的操作都会在这一个dc上进行</span>
</span><span class='line'>      <span class="nx">tx</span> <span class="p">=</span> <span class="o">&amp;</span><span class="nx">Tx</span><span class="p">{</span>
</span><span class='line'>          <span class="nx">db</span><span class="p">:</span>          <span class="nx">db</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">dc</span><span class="p">:</span>          <span class="nx">dc</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">releaseConn</span><span class="p">:</span> <span class="nx">release</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">txi</span><span class="p">:</span>         <span class="nx">txi</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">cancel</span><span class="p">:</span>      <span class="nx">cancel</span><span class="p">,</span>
</span><span class='line'>          <span class="nx">ctx</span><span class="p">:</span>         <span class="nx">ctx</span><span class="p">,</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">go</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">awaitDone</span><span class="p">()</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">tx</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>执行阶段，exec和query过程类似，下面只分析exec的过程</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'>  <span class="kd">func</span> <span class="p">(</span><span class="nx">tx</span> <span class="o">*</span><span class="nx">Tx</span><span class="p">)</span> <span class="nx">QueryContext</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">query</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">args</span> <span class="o">...</span><span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="o">*</span><span class="nx">Rows</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="c1">// 取出之前保存在tx上的连接dc</span>
</span><span class='line'>      <span class="nx">dc</span><span class="p">,</span> <span class="nx">release</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">grabConn</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 其实就是db.queryDC的逻辑</span>
</span><span class='line'>      <span class="c1">// 现在知道为什么tx上要记一个db对象了</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">db</span><span class="p">.</span><span class="nx">queryDC</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">dc</span><span class="p">,</span> <span class="nx">release</span><span class="p">,</span> <span class="nx">query</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="kd">func</span> <span class="p">(</span><span class="nx">tx</span> <span class="o">*</span><span class="nx">Tx</span><span class="p">)</span> <span class="nx">grabConn</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="nx">driverConn</span><span class="p">,</span> <span class="nx">releaseConn</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">default</span><span class="p">:</span>
</span><span class='line'>      <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Err</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>      <span class="c1">// closeme.RLock must come before the check for isDone to prevent the Tx from</span>
</span><span class='line'>      <span class="c1">// closing while a query is executing.</span>
</span><span class='line'>      <span class="c1">// 这个读写锁的目的是保证当tx被关闭时已经没有任何其它的查询在进行了</span>
</span><span class='line'>      <span class="c1">// 注意到一般每一个资源对象都有这样一个锁，比如Row、Stmt</span>
</span><span class='line'>      <span class="nx">tx</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RLock</span><span class="p">()</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">isDone</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>          <span class="nx">tx</span><span class="p">.</span><span class="nx">closemu</span><span class="p">.</span><span class="nx">RUnlock</span><span class="p">()</span>
</span><span class='line'>          <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ErrTxDone</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="k">if</span> <span class="nx">hookTxGrabConn</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="c1">// test hook</span>
</span><span class='line'>          <span class="nx">hookTxGrabConn</span><span class="p">()</span>
</span><span class='line'>      <span class="p">}</span>
</span><span class='line'>      <span class="c1">// 返回的是tx上记的那个dc</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">dc</span><span class="p">,</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">closemuRUnlockRelease</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>参考</h3>

<ul>
<li><a href="https://chenjiayang.me/2019/08/10/go-sql-database-pool/">借 Go 语言 database/sql 包谈数据库驱动和连接池设计</a></li>
<li><a href="http://luodw.cc/2016/08/28/golang02/">golang之database/sql与go-sql-driver</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[job queue in Golang]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/07/20/job-queue-in-golang/"/>
    <updated>2019-07-20T12:16:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/07/20/job-queue-in-golang</id>
    <content type="html"><![CDATA[<p>异步任务，还是蛮常见的</p>

<h3>算不上 job queue的形式</h3>

<p><code>go process(job)</code></p>

<p>这其实算不上一个queue，但简单。同时，少了一些对并发的控制，比如控制同时执行的任务数等。</p>

<h3>最简单的 job queue</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="nx">job</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">jobChan</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// make a channel with a capacity of 100.</span>
</span><span class='line'><span class="nx">jobChan</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// start the worker</span>
</span><span class='line'><span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// enqueue a job</span>
</span><span class='line'><span class="nx">jobChan</span> <span class="o">&lt;-</span> <span class="nx">job</span>
</span></code></pre></td></tr></table></div></figure>


<p>注释已经很清楚了，通过向channel发消息来提交任务，worker 从 channel 中取任务做。注意 jobChan 是一个固定长度的 channel，这能够实现 producer throtting，当 queue 中已经有100个 task 时，此时的 enqueue 操作会阻塞。</p>

<h3>非阻塞式 enqueue</h3>

<p>如果 enqueue 时不想阻塞呢？比如我想如果队列满了，就直接给 client 端返回失败，告诉它等会再试试。可以这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// TryEnqueue tries to enqueue a job to the given job channel. Returns true if</span>
</span><span class='line'><span class="c1">// the operation was successful, and false if enqueuing would not have been</span>
</span><span class='line'><span class="c1">// possible without blocking. Job is not enqueued in the latter case.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">TryEnqueue</span><span class="p">(</span><span class="nx">job</span> <span class="nx">Job</span><span class="p">,</span> <span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">case</span> <span class="nx">jobChan</span> <span class="o">&lt;-</span> <span class="nx">job</span><span class="p">:</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">true</span>
</span><span class='line'>    <span class="k">default</span><span class="p">:</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// then you can do this</span>
</span><span class='line'><span class="k">if</span> <span class="p">!</span><span class="nx">TryEnqueue</span><span class="p">(</span><span class="nx">job</span><span class="p">,</span> <span class="kd">chan</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="s">&quot;max capacity reached&quot;</span><span class="p">,</span> <span class="mi">503</span><span class="p">)</span>
</span><span class='line'>    <span class="k">return</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>停止 worker</h3>

<p>如果没有任务需要做了，那么可以：</p>

<p><code>close(jobChan)</code></p>

<p>因为 worker 是通过<code>for job := range jobChan {...}</code> 这种形式来取任务的，当 channel 被关闭后，for loop 会停止循环，继而结束 worker。</p>

<p>需要注意的是：即使 channel 被 close 的时候，channel 里还有尚未被消费的 task，这些 task 照样会被正常消费完</p>

<h3>等待 worker 退出</h3>

<p><code>close</code> channel 只会通知 worker 当前已无更多任务，但并不会等待 worker 把任务做完，所以我们需要一种等待 worker 的机制：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// use a WaitGroup </span>
</span><span class='line'><span class="kd">var</span> <span class="nx">wg</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">WaitGroup</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">defer</span> <span class="nx">wg</span><span class="p">.</span><span class="nx">Done</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">for</span> <span class="nx">job</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">jobChan</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// increment the WaitGroup before starting the worker</span>
</span><span class='line'><span class="nx">wg</span><span class="p">.</span><span class="nx">Add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'><span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// to stop the worker, first close the job channel</span>
</span><span class='line'><span class="nb">close</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// then wait using the WaitGroup</span>
</span><span class='line'><span class="nx">wg</span><span class="p">.</span><span class="nx">Wait</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<h3>带超时时间的等待</h3>

<p>如果 worker 的任务一直没有做完，那么<code>wg.Wait()</code> 会无休止的等待下去，如果我们无法承受一直等待怎么办呢？</p>

<p>可以把<code>wg.Wait()</code>封装一下，给它增加 <code>timeout</code> 的功能</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// WaitTimeout does a Wait on a sync.WaitGroup object but with a specified</span>
</span><span class='line'><span class="c1">// timeout. Returns true if the wait completed without timing out, false</span>
</span><span class='line'><span class="c1">// otherwise.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">WaitTimeout</span><span class="p">(</span><span class="nx">wg</span> <span class="o">*</span><span class="nx">sync</span><span class="p">.</span><span class="nx">WaitGroup</span><span class="p">,</span> <span class="nx">timeout</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Duration</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">ch</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span>
</span><span class='line'>    <span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">wg</span><span class="p">.</span><span class="nx">Wait</span><span class="p">()</span>
</span><span class='line'>        <span class="nb">close</span><span class="p">(</span><span class="nx">ch</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}()</span>
</span><span class='line'>    <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ch</span><span class="p">:</span>
</span><span class='line'>            <span class="k">return</span> <span class="kc">true</span>
</span><span class='line'>    <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">time</span><span class="p">.</span><span class="nx">After</span><span class="p">(</span><span class="nx">timeout</span><span class="p">):</span>
</span><span class='line'>            <span class="k">return</span> <span class="kc">false</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// now use the WaitTimeout instead of wg.Wait()</span>
</span><span class='line'><span class="nx">WaitTimeout</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">wg</span><span class="p">,</span> <span class="mi">5</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p></p>

<h3>取消 worker</h3>

<p>上面的代码中，如果我们发出退出的信号，worker 们会做完当前正在做的任务然后再退出，如果我们想让它立即退出该怎么办呢？</p>

<p>可以利用<code>context.Context</code></p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// create a context that can be cancelled</span>
</span><span class='line'><span class="nx">ctx</span><span class="p">,</span> <span class="nx">cancel</span> <span class="o">:=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">WithCancel</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">())</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// start the goroutine passing it the context</span>
</span><span class='line'><span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>            <span class="k">return</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">case</span> <span class="nx">job</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">jobChan</span><span class="p">:</span>
</span><span class='line'>            <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Invoke cancel when the worker needs to be stopped. This *does not* wait</span>
</span><span class='line'><span class="c1">// for the worker to exit.</span>
</span><span class='line'><span class="nx">cancel</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>但这里有一个小坑，当在收到退出信号时，同时也有job可取，那么 select 会<strong>随机</strong>选择一个路径来执行，并不会优先现在退出的路径，如果你想优先退出的话，需要这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">var</span> <span class="nx">flag</span> <span class="kt">uint64</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>            <span class="k">return</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">case</span> <span class="nx">job</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">jobChan</span><span class="p">:</span>
</span><span class='line'>            <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">atomic</span><span class="p">.</span><span class="nx">LoadUint64</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">flag</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="p">{</span>
</span><span class='line'>                <span class="k">return</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// set the flag first, before cancelling</span>
</span><span class='line'><span class="nx">atomic</span><span class="p">.</span><span class="nx">StoreUint64</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">flag</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
</span><span class='line'><span class="nx">cancel</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>或者这样：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">ctx</span><span class="p">.</span><span class="nx">Done</span><span class="p">():</span>
</span><span class='line'>            <span class="k">return</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">case</span> <span class="nx">job</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">jobChan</span><span class="p">:</span>
</span><span class='line'>            <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>            <span class="k">if</span> <span class="nx">ctx</span><span class="p">.</span><span class="nx">Err</span><span class="p">()</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>                <span class="k">return</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="nx">cancel</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>(译者注：我觉得这样仍然不能保证退出路径优先被执行呀）</p>

<h3>不用 context 也能取消 worker</h3>

<p>如下，在某些场景下可能写法还更简洁，当然原理是一样的</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// create a cancel channel</span>
</span><span class='line'><span class="nx">cancelChan</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// start the goroutine passing it the cancel channel </span>
</span><span class='line'><span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">,</span> <span class="nx">cancelChan</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="nx">Job</span><span class="p">,</span> <span class="nx">cancelChan</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">for</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">select</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">case</span> <span class="o">&lt;-</span><span class="nx">cancelChan</span><span class="p">:</span>
</span><span class='line'>            <span class="k">return</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">case</span> <span class="nx">job</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">jobChan</span><span class="p">:</span>
</span><span class='line'>            <span class="nx">process</span><span class="p">(</span><span class="nx">job</span><span class="p">)</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// to cancel the worker, close the cancel channel</span>
</span><span class='line'><span class="nb">close</span><span class="p">(</span><span class="nx">cancelChan</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h3>worker 池</h3>

<p>最简单的就是启动多个 worker，让它们读取同一个 channel</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">for</span> <span class="nx">i</span><span class="o">:=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">i</span><span class="p">&lt;</span><span class="nx">workerCount</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>如果想要等待 worker 退出</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">for</span> <span class="nx">i</span><span class="o">:=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">i</span><span class="p">&lt;</span><span class="nx">workerCount</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">wg</span><span class="p">.</span><span class="nx">Add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// wait for all workers to exit</span>
</span><span class='line'><span class="nx">wg</span><span class="p">.</span><span class="nx">Wait</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>取消 worker</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// create cancel channel</span>
</span><span class='line'><span class="nx">cancelChan</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// pass the channel to the workers, let them wait on it</span>
</span><span class='line'><span class="k">for</span> <span class="nx">i</span><span class="o">:=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">i</span><span class="p">&lt;</span><span class="nx">workerCount</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">jobChan</span><span class="p">,</span> <span class="nx">cancelChan</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// close the channel to signal the workers</span>
</span><span class='line'><span class="nb">close</span><span class="p">(</span><span class="nx">cancelChan</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h3>参考</h3>

<ul>
<li><a href="https://disqus.com/embed/comments/?base=default&amp;f=opsdash&amp;t_u=https%3A%2F%2Fwww.opsdash.com%2Fblog%2Fjob-queues-in-go.html&amp;t_d=Job%20Queues%20in%20Go%20-%20OpsDash&amp;t_t=Job%20Queues%20in%20Go%20-%20OpsDash&amp;s_o=default#version=5c281b90be9cbae86fbebcbaed6c8c9b">JOB QUEUES IN GO</a></li>
<li><a href="https://gist.github.com/harlow/dbcd639cf8d396a2ab73">Golang Workers / Job Queue</a></li>
<li><a href="http://nesv.github.io/golang/2014/02/25/worker-queues-in-go.html">Writing worker queues, in Go</a></li>
<li><a href="http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/">Handling 1 Million Requests per Minute with Go</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Golang 中的文字编码问题]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/05/19/a-encoding-problem-in-golang/"/>
    <updated>2019-05-19T22:41:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/05/19/a-encoding-problem-in-golang</id>
    <content type="html"><![CDATA[<h3>过程描述</h3>

<p>记录一个文字编码的问题</p>

<p>在请求支付宝支付的接口时，发现返回的Response如果有中文的话，print出来后会有乱码。第一反应肯定是编码有问题，那就先转一下编码吧。</p>

<p>一开始的代码大概如下：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">resp</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">Post</span><span class="p">(</span><span class="nx">aliPayBillURL</span><span class="p">,</span> <span class="s">&quot;application/x-www-form-urlencoded&quot;</span><span class="p">,</span> <span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBufferString</span><span class="p">(</span><span class="nx">output</span><span class="p">))</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="k">defer</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'><span class="nx">body</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ioutil</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="c1">//...</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">queryResp</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">data</span> <span class="o">:=</span> <span class="nx">queryResp</span><span class="p">.</span><span class="nx">Data</span>
</span><span class='line'><span class="c1">// data.SubMsg里含有中文，会乱码</span>
</span><span class='line'><span class="k">if</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Code</span> <span class="o">!=</span> <span class="s">&quot;10000&quot;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;download failed: msg: %s, sub_code: %s, sub_msg: %s\n&quot;</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Msg</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubCode</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubMsg</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;download failed&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>改为了：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">resp</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">Post</span><span class="p">(</span><span class="nx">aliPayBillURL</span><span class="p">,</span> <span class="s">&quot;application/x-www-form-urlencoded&quot;</span><span class="p">,</span> <span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBufferString</span><span class="p">(</span><span class="nx">output</span><span class="p">))</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="k">defer</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'><span class="nx">body</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ioutil</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="c1">//...</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">body</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">queryResp</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">data</span> <span class="o">:=</span> <span class="nx">queryResp</span><span class="p">.</span><span class="nx">Data</span>
</span><span class='line'><span class="nx">converted</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">gbkToUtf8</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">SubMsg</span><span class="p">))</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="k">if</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Code</span> <span class="o">!=</span> <span class="s">&quot;10000&quot;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;download failed: msg: %s, sub_code: %s, sub_msg: %s\n&quot;</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Msg</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubCode</span><span class="p">,</span> <span class="nx">converted</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;download failed&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>结果是依旧乱码。。难道不是编码的问题？</p>

<p>改<code>%s</code>为<code>%x</code>看看具体字节：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">if</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Code</span> <span class="o">!=</span> <span class="s">&quot;10000&quot;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;download failed: msg: %s, sub_code: %s, sub_msg: %x\n&quot;</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Msg</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubCode</span><span class="p">,</span> <span class="nx">converted</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;download failed&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>然后看到了这个<code>efbfbdefbfbdd0a7efbfbdefbfbd 4170704944 efbfbdefbfbdefbfbd</code>，之所以分成了三段贴出来，是因为我发现中间那段对应了部分正常显示的字符“Appid”，然后尝试手动解码一下其它两段，不管是GBK还是utf8都没有找到对应的字符。</p>

<p>然后发现第一段和最后一段的模式有点类似啊，都是<code>efbfbd</code>啥的，有点奇怪。尝试直接搜索<code>efbfbd</code>，定位到一篇<a href="https://liudanking.com/golang/utf-8_replacement_character/">文章</a>，里面主要讲到两点：</p>

<ul>
<li>这个序列就是utf8编码的，但它比较特殊，它是专门用来替换那些在解码时不认识的码点的，显示出来就是「�」</li>
<li>某些语言，比如 Goang，会自动进行这个转换，不会报错</li>
</ul>


<p>借用一下原文的例子：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">now</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">().</span><span class="nx">Unix</span><span class="p">()</span> <span class="c1">// 一个无效的码点值</span>
</span><span class='line'><span class="nx">str</span> <span class="o">:=</span> <span class="nb">string</span><span class="p">(</span><span class="nx">now</span><span class="p">)</span> <span class="c1">// golang是utf-8编码，会对无效码点进行替换</span>
</span><span class='line'><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;%X&quot;</span><span class="p">,</span> <span class="p">[]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">str</span><span class="p">))</span> <span class="c1">// EFBFBD，即字符「�」</span>
</span></code></pre></td></tr></table></div></figure>


<p>这一下坚定了认为是编码的问题，那就是转码出问题了，再一看，我是在json decode之后再转码的，json decode的时候可能已经把数据按utf8解码了，而数据如果不是utf8编码的话，那就都被替换成<code>efbfbd</code>了，改为在json decode之前就转码试试：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">resp</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">client</span><span class="p">.</span><span class="nx">Post</span><span class="p">(</span><span class="nx">aliPayBillURL</span><span class="p">,</span> <span class="s">&quot;application/x-www-form-urlencoded&quot;</span><span class="p">,</span> <span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBufferString</span><span class="p">(</span><span class="nx">output</span><span class="p">))</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="k">defer</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span>
</span><span class='line'><span class="nx">body</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ioutil</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">converted</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">gbkToUtf8</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="nx">body</span><span class="p">))</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nb">string</span><span class="p">(</span><span class="nx">converted</span><span class="p">))</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">converted</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">queryResp</span><span class="p">)</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">data</span> <span class="o">:=</span> <span class="nx">queryResp</span><span class="p">.</span><span class="nx">Data</span>
</span><span class='line'><span class="k">if</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Code</span> <span class="o">!=</span> <span class="s">&quot;10000&quot;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">&quot;download failed: msg: %s, sub_code: %s, sub_msg: %s\n&quot;</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Msg</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubCode</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">SubMsg</span><span class="p">)</span>
</span><span class='line'>    <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;download failed&quot;</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>这次没问题了。</p>

<h3>反思 &amp;&amp; 教训</h3>

<ul>
<li>获取到数据后不要做任何操作，先做转码</li>
<li>记住<code>efbfbd</code>，它是一个信号，说明源数据一定不是utf8编码的</li>
</ul>


<p>最后，记录一下 Golang 中如何从 GBK 转为 utf8</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kn">import</span> <span class="p">(</span>
</span><span class='line'><span class="nx">golang</span><span class="p">.</span><span class="nx">org</span><span class="o">/</span><span class="nx">x</span><span class="o">/</span><span class="nx">text</span><span class="o">/</span><span class="nx">encoding</span><span class="o">/</span><span class="nx">simplifiedchinese</span>
</span><span class='line'><span class="nx">golang</span><span class="p">.</span><span class="nx">org</span><span class="o">/</span><span class="nx">x</span><span class="o">/</span><span class="nx">text</span><span class="o">/</span><span class="nx">transform</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">gbkToUtf8</span><span class="p">(</span><span class="nx">s</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">)</span> <span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">reader</span> <span class="o">:=</span> <span class="nx">transform</span><span class="p">.</span><span class="nx">NewReader</span><span class="p">(</span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewReader</span><span class="p">(</span><span class="nx">s</span><span class="p">),</span> <span class="nx">simplifiedchinese</span><span class="p">.</span><span class="nx">GBK</span><span class="p">.</span><span class="nx">NewDecoder</span><span class="p">())</span>
</span><span class='line'>    <span class="nx">d</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">ioutil</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">reader</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">d</span><span class="p">,</span> <span class="kc">nil</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>参考</h3>

<ul>
<li><a href="https://liudanking.com/golang/utf-8_replacement_character/">你应该记住的一个UTF-8字符「EF BF BD」</a></li>
<li><a href="http://mengqi.info/html/2015/201507071345-using-golang-to-convert-text-between-gbk-and-utf-8.html">Golang 中的 UTF-8 与 GBK 编码转换</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[database with sqlx in Golang]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/05/09/database-with-sqlx-in-golang/"/>
    <updated>2019-05-09T10:39:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/05/09/database-with-sqlx-in-golang</id>
    <content type="html"><![CDATA[<h2>database with sqlx in Golang</h2>

<h3>Parameterized Queries</h3>

<p>翻译为参数化查询？是个什么概念呢？</p>

<p>Bind parameters—also called dynamic parameters or bind variables—are an alternative way to pass data to the database. Instead of putting the values directly into the SQL statement, you just use a placeholder like ?, :name or @name and provide the actual values using a separate API call.</p>

<p>原来说的是这种形式：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Query</span><span class="p">(</span><span class="s">&quot;select * from users where name = ?&quot;</span><span class="p">,</span> <span class="s">&quot;linuxfish&quot;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>与之比较的是还有一种写法，这种是臭名昭著的sql拼接，之前一直理解上面的那种写法也是sql拼接，看来是错怪它了：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// 这样写是有被sql注入的危险的</span>
</span><span class='line'><span class="nx">myName</span> <span class="o">:=</span> <span class="nx">getNameFromUser</span><span class="p">()</span>
</span><span class='line'><span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Query</span><span class="p">(</span><span class="s">&quot;select * from users where name = &quot;</span> <span class="o">+</span> <span class="nx">myName</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>那么 Parameterized Queries 的好处是什么？</p>

<ul>
<li>安全

<ul>
<li>the best way to prevent SQL injection.</li>
</ul>
</li>
<li>性能高

<ul>
<li>数据库会缓存执行计划，但必须是一模一样的sql，差一点都不行，那么在实际应用中这个缓存的作用就大打折扣了，因为<code>select * from users where id = 2</code>和<code>select * from users where id = 3</code>在数据库看来也是不一样的，尽管它们本质上是一个sql。</li>
<li>用Parameterized Queries可以解决这个问题，因为变化的值变成参数了，对于数据库而言，请求一直是<code>select * from users where id = ?</code></li>
<li>但其实并没有那么完美，使用Parameterized Queries后，尽管sql不变了，但相对于真正执行的sql，它缺失了一部分信息，导致优化器无法作出最优的执行计划，因为参数的不同，最终的执行计划也可能是不一样的，因此，使用Parameterized Queries会性能高，是个伪命题</li>
<li>这又是个取舍的问题（tradeoff），从总体来看，使用Parameterized Queries的好处还是大于坏处，所以，尽量还是使用Parameterized Queries</li>
<li>参考：<a href="https://use-the-index-luke.com/sql/where-clause/bind-parameters">Parameterized Queries</a></li>
</ul>
</li>
</ul>


<h3>what sqlx brings us</h3>

<h4>StructScan</h4>

<p>这是下面说的 Get 和 Select 实现的基础，支持把select出来的字段一下scan进一个struct</p>

<p>The primary extension on sqlx.Rows is StructScan(), which automatically scans results into struct fields. Note that the fields must be exported (capitalized) in order for sqlx to be able to write into them, something true of all marshallers in Go. You can use the db struct tag to specify which column name maps to each struct field, or set a new default mapping with db.MapperFunc(). The default behavior is to use strings.Lower on the field name to match against the column names.</p>

<p>那么是根据什么规则scan的呢？如上面所说，优先会找字段有没有设置<code>db</code> tag，如果设置了就用它，如果没有设置，那么默认使用的是<code>strings.Lower(fieldName)</code>，当然这个转换机制是可以定制的，你可以传入自己的处理函数。对应的实现代码应该在这里：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">var</span> <span class="nx">NameMapper</span> <span class="p">=</span> <span class="nx">strings</span><span class="p">.</span><span class="nx">ToLower</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// ...</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// mapper returns a valid mapper using the configured NameMapper func.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">mapper</span><span class="p">()</span> <span class="o">*</span><span class="nx">reflectx</span><span class="p">.</span><span class="nx">Mapper</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">mprMu</span><span class="p">.</span><span class="nx">Lock</span><span class="p">()</span>
</span><span class='line'>    <span class="k">defer</span> <span class="nx">mprMu</span><span class="p">.</span><span class="nx">Unlock</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="nx">mpr</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">mpr</span> <span class="p">=</span> <span class="nx">reflectx</span><span class="p">.</span><span class="nx">NewMapperFunc</span><span class="p">(</span><span class="s">&quot;db&quot;</span><span class="p">,</span> <span class="nx">NameMapper</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="nx">origMapper</span> <span class="o">!=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">NameMapper</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="c1">// if NameMapper has changed, create a new mapper</span>
</span><span class='line'>        <span class="nx">mpr</span> <span class="p">=</span> <span class="nx">reflectx</span><span class="p">.</span><span class="nx">NewMapperFunc</span><span class="p">(</span><span class="s">&quot;db&quot;</span><span class="p">,</span> <span class="nx">NameMapper</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">origMapper</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">NameMapper</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">mpr</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// 定制自己的mapper</span>
</span><span class='line'><span class="kn">import</span> <span class="s">&quot;github.com/jmoiron/sqlx/reflectx&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Create a new mapper which will use the struct field tag &quot;json&quot; instead of &quot;db&quot;</span>
</span><span class='line'><span class="nx">db</span><span class="p">.</span><span class="nx">Mapper</span> <span class="p">=</span> <span class="nx">reflectx</span><span class="p">.</span><span class="nx">NewMapperFunc</span><span class="p">(</span><span class="s">&quot;json&quot;</span><span class="p">,</span> <span class="nx">strings</span><span class="p">.</span><span class="nx">ToLower</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h4>Get and Select</h4>

<p>把 query 和 scan 结合到一步了！</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">p</span> <span class="o">:=</span> <span class="nx">Place</span><span class="p">{}</span>
</span><span class='line'><span class="nx">pp</span> <span class="o">:=</span> <span class="p">[]</span><span class="nx">Place</span><span class="p">{}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// this will pull the first place directly into p</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">p</span><span class="p">,</span> <span class="s">&quot;SELECT * FROM place LIMIT 1&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// this will pull places with telcode &gt; 50 into the slice pp</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Select</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">pp</span><span class="p">,</span> <span class="s">&quot;SELECT * FROM place WHERE telcode &gt; ?&quot;</span><span class="p">,</span> <span class="mi">50</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// they work with regular types as well</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">id</span> <span class="kt">int</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">id</span><span class="p">,</span> <span class="s">&quot;SELECT count(*) FROM place&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// fetch at most 10 place names</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">names</span> <span class="p">[]</span><span class="kt">string</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Select</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">names</span><span class="p">,</span> <span class="s">&quot;SELECT name FROM place LIMIT 10&quot;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>risks 潜在风险点</p>

<p>Select can save you a lot of typing, but beware! It&rsquo;s semantically different from Queryx, since <strong>it will load the entire result set into memory at once.</strong> If that set is not bounded by your query to some reasonable size, it might be best to use the classic Queryx/StructScan iteration instead. 作者建议如果返回的数据集大小不确定，还是使用经典的for Next Scan的模式</p>

<h5>Todo： 了解下 Get 和 Select 的原理</h5>

<h4>Transactions</h4>

<p>Exec and all other query verbs will ask the DB for a connection and then return it to the pool each time. There&rsquo;s no guarantee that you will receive the same connection that the BEGIN statement was executed on. To use transactions, you must therefore use <code>DB.Begin()</code> 必须先 Begin 一个transaction</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">tx</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Begin</span><span class="p">()</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">Exec</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">tx</span><span class="p">.</span><span class="nx">Commit</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>Since transactions are connection state, the Tx object must bind and control a single connection from the pool. A Tx will maintain that single connection for its entire life cycle, releasing it only when Commit() or Rollback() is called. You should take care to call at least one of these, or else the connection will be held until garbage collection. 只有当<code>Commit()</code>或<code>Rollback()</code>时数据库连接才被放回连接池，记得调这两个函数，否则会有连接不释放的问题</p>

<p>Because you only have one connection to use in a transaction, you can only execute one statement at a time; the cursor types Row and Rows must be Scanned or Closed, respectively, before executing another query. If you attempt to send the server data while it is sending you a result, it can potentially corrupt the connection. 事务中只有一个连接可以使用，因此查询（Query）必须被Scan完或者Close后，后续的查询才能继续执行。</p>

<h4>Named Queries</h4>

<p>Named queries are common to many other database packages. They allow you to use a bindvar syntax which refers to the names of struct fields or map keys to bind variables a query, rather than having to refer to everything positionally. 这个类似于 Python format中的这种写法：<code>print("网站名：{name}, 地址 {url}".format(name="菜鸟教程", url="www.runoob.com"))</code> 好处是不用在意参数的位置了。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="c1">// named query with a struct</span>
</span><span class='line'><span class="nx">p</span> <span class="o">:=</span> <span class="nx">Place</span><span class="p">{</span><span class="nx">Country</span><span class="p">:</span> <span class="s">&quot;South Africa&quot;</span><span class="p">}</span>
</span><span class='line'><span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">NamedQuery</span><span class="p">(</span><span class="s">`SELECT * FROM place WHERE country=:country`</span><span class="p">,</span> <span class="nx">p</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// named query with a map</span>
</span><span class='line'><span class="nx">m</span> <span class="o">:=</span> <span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kd">interface</span><span class="p">{}{</span><span class="s">&quot;city&quot;</span><span class="p">:</span> <span class="s">&quot;Johannesburg&quot;</span><span class="p">}</span>
</span><span class='line'><span class="nx">result</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">NamedExec</span><span class="p">(</span><span class="s">`SELECT * FROM place WHERE city=:city`</span><span class="p">,</span> <span class="nx">m</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>Named query support is implemented by parsing the query for the :param syntax and replacing it with the bindvar supported by the underlying database, then performing the mapping at execution, so it is usable on any database that sqlx supports. 实现原理</p>

<h4>Advanced Scanning</h4>

<h5>Scan Destination Safety</h5>

<p>扫描的destination变量里的字段必须是select的字段的超集，否则会报错，当然也可以忽略这个错误</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">var</span> <span class="nx">p</span> <span class="nx">Person</span>
</span><span class='line'><span class="c1">// err here is not nil because there are no field destinations for columns in `place`</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">p</span><span class="p">,</span> <span class="s">&quot;SELECT * FROM person, place LIMIT 1;&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// this will NOT return an error, even though place columns have no destination</span>
</span><span class='line'><span class="nx">udb</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Unsafe</span><span class="p">()</span>
</span><span class='line'><span class="nx">err</span> <span class="p">=</span> <span class="nx">udb</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">p</span><span class="p">,</span> <span class="s">&quot;SELECT * FROM person, place LIMIT 1;&quot;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<h5>Alternate Scan Types</h5>

<p>In addition to using Scan and StructScan, an sqlx Row or Rows can be used to automatically return a slice or a map of results，还能scan成slice和map！！:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Queryx</span><span class="p">(</span><span class="s">&quot;SELECT * FROM place&quot;</span><span class="p">)</span>
</span><span class='line'><span class="k">for</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// cols is an []interface{} of all of the column results</span>
</span><span class='line'>    <span class="nx">cols</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">SliceScan</span><span class="p">()</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="nx">rows</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">Queryx</span><span class="p">(</span><span class="s">&quot;SELECT * FROM place&quot;</span><span class="p">)</span>
</span><span class='line'><span class="k">for</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">results</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kd">interface</span><span class="p">{})</span>
</span><span class='line'>    <span class="nx">err</span> <span class="p">=</span> <span class="nx">rows</span><span class="p">.</span><span class="nx">MapScan</span><span class="p">(</span><span class="nx">results</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h5>Custom Types / 自定义类型</h5>

<p>the examples above all used the built-in types to both scan and query with, but database/sql provides interfaces to allow you to use any custom types. database/sql包提供了接口让你来扩展你的自定义类型，让它们也可以被Scan和Query</p>

<p>参考：<a href="http://jmoiron.net/blog/built-in-interfaces">Built In Interfaces</a></p>

<h4>The Connection Pool</h4>

<p>database/sql 包内置连接池，而且提供了几个函数来定制连接池的行为：</p>

<ul>
<li>DB.SetMaxIdleConns(n int)

<ul>
<li> sets the maximum number of connections in the idle connection pool</li>
<li> The default max idle connections is currently 2. This may change in a future release. 默认2个</li>
</ul>
</li>
<li>DB.SetMaxOpenConns(n int)

<ul>
<li>sets the maximum number of open connections to the database.</li>
<li>默认无限制</li>
</ul>
</li>
<li>DB.SetConnMaxLifetime(d time.Duration)

<ul>
<li>sets the maximum amount of time a connection may be reused.</li>
</ul>
</li>
</ul>


<h3>参考</h3>

<ul>
<li><a href="http://jmoiron.github.io/sqlx/">Illustrated guide to SQLX</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Error Handling in Go]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/05/09/error-handling-in-go/"/>
    <updated>2019-05-09T08:50:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/05/09/error-handling-in-go</id>
    <content type="html"><![CDATA[<p>From: <a href="https://dave.cheney.net/2016/04/27/dont-just-check-errors-handle-them-gracefully">Don’t just check errors, handle them gracefully</a></p>

<h3>error变量</h3>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">var</span> <span class="nx">ErrNotFound</span> <span class="p">=</span> <span class="nx">errors</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">&quot;datastore:notfound&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nx">err</span> <span class="o">:=</span> <span class="nx">Foo</span><span class="p">()</span>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">ErrNotFound</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// ...</span>
</span><span class='line'>    <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>优点：简单</p>

<p>缺点：不能带上下文，只有一个简单的固定字符描述；而且错误要耦合进调用方</p>

<h3>error类型</h3>

<p>好处是可以承载更多的上下文信息</p>

<p>缺点还是这个错误必须耦合进调用方</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">type</span> <span class="nx">PathError</span> <span class="kd">struct</span> <span class="p">{</span>
</span><span class='line'><span class="nx">Op</span> <span class="kt">string</span>
</span><span class='line'><span class="nx">Path</span> <span class="kt">string</span>
</span><span class='line'><span class="nx">Err</span> <span class="kt">error</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">PathError</span><span class="p">)</span> <span class="nx">Error</span><span class="p">()</span> <span class="kt">string</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="s">&quot;&quot;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="k">if</span> <span class="nx">err</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">PathError</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span>
</span><span class='line'>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>opaque error handling / 黑盒错误处理</h3>

<p>不对返回的error做任何假设，因此也不能干啥。。  直接返回error</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">foo</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">x</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">bar</span><span class="p">()</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">err</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// use x</span>
</span></code></pre></td></tr></table></div></figure>


<h3>assert errors for behavior, not type / 根据行为来处理</h3>

<p>上面的方法太简单。。有时候确实是需要区分error种类，来按情况处理的。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">type</span> <span class="nx">temporary</span> <span class="kd">interface</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">Temporary</span><span class="p">()</span> <span class="kt">bool</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// IsTemporary returns true if err is temporary.</span>
</span><span class='line'><span class="kd">func</span> <span class="nx">IsTemporary</span><span class="p">(</span><span class="nx">err</span> <span class="kt">error</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">te</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">temporary</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span> <span class="nx">ok</span> <span class="o">&amp;&amp;</span> <span class="nx">te</span><span class="p">.</span><span class="nx">Temporary</span><span class="p">()</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>优点：调用方不用依赖定义error的包了</p>

<p>缺点：这种方法依赖 第三方包作者的配合，需要他先让他返回的error满足一定的interface（比如temporary)，包的使用者才能这么用呢</p>

<p>Don’t assert errors for type, assert for behaviour.</p>

<p>For package authors, if your package generates errors of a temporary nature, ensure you return error types that implement the respective interface methods. If you wrap error values on the way out, ensure that your wrappers respect the interface(s) that the underlying error value implemented.</p>

<p>For package users, if you need to inspect an error, use interfaces to assert the behaviour you expect, not the error’s type. Don’t ask package authors for public error types; ask that they make their types conform to common interfaces by supplying Timeout() or Temporary() methods as appropriate.</p>

<p>参考：</p>

<ul>
<li><a href="https://dave.cheney.net/2014/12/24/inspecting-errors">Inspecting errors</a></li>
</ul>


<h3>extra tip</h3>

<ul>
<li><p>annotating errors</p></li>
<li><p>only handle errors once</p></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Redis中set元素设置过期？]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/03/24/expire-elements-in-redis-set/"/>
    <updated>2019-03-24T21:58:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/03/24/expire-elements-in-redis-set</id>
    <content type="html"><![CDATA[<h3>需求</h3>

<p>项目中遇到一个需求，要实现一个黑白名单的服务，对外暴露向名单添加元素和检查元素是否在名单中的接口，这个用 Redis 实现是很容易的，有现成的类型支持，但还有个
需求是名单中的某些元素要支持自动过期，即过了一定时间后，元素自动从名单中消失了，再去 check 该元素是否在名单时，应该返回 false。Redis 是支持过期的，但它只
支持 key 级别的过期。</p>

<h3>方案</h3>

<p>看了下，早有人给 Redis 项目提出类似的issue：要求支持元素级别的过期。项目的维护者也早已指出：不可能支持这样的 feature，因为违背了 Redis 的设计理念：简单、高效。
不过，在 Google Group 上看到 Redis 的作者针对这类需求给出了3个实现方案：</p>

<ul>
<li><p>用 redis 的普通 set 类型实现。把时间戳 encode 进元素名称中，比如平常只是 add 一个元素 foo，现在需要 add 元素名：<code>foo:&lt;timestamp&gt;</code>。那么每次需要 check 这个元素
的时候先获取一下当前的时间戳跟保存的时间戳比较一下，如果已经过期，则删除它。这个方案的缺点是：如果 add 了一个元素后，一直不再访问它，那么尽管给它设置了过期时间，
那么它还是会一直存在。</p></li>
<li><p>使用 redis 的 sorted set来实现。score 是元素过期的时间戳，value 是元素名。在代码中每秒执行一次<code>zremrangebyscore</code>来清除已过期的元素。</p></li>
<li><p>第三种没看懂。。</p></li>
</ul>


<p>在 stackoverflow 上看到有人给出了类似上面说的第二种方案的具体实现，参考文末链接。</p>

<h3>参考</h3>

<ul>
<li><a href="https://groups.google.com/forum/#!topic/redis-db/rXXMCLNkNSs">Pattern for expiring set members</a></li>
<li><a href="https://stackoverflow.com/questions/11810020/how-to-handle-session-expire-basing-redis/11815594#11815594">how to handle session expire basing redis?</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Golang FAQ]]></title>
    <link href="http://cs50Mu.github.io/blog/2019/02/01/golang-faq/"/>
    <updated>2019-02-01T22:37:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2019/02/01/golang-faq</id>
    <content type="html"><![CDATA[<p>这几天看了下Golang FAQ，解了不少困惑，我相信有人会有同样的困惑，挑选了一些有意思的问题翻译了一下。</p>

<h3>Design</h3>

<blockquote><p>Does Go have a runtime? / Go有运行时吗？</p></blockquote>

<p>Go确实有一个runtime库(runtime library)。这个runtime库主要实现了垃圾回收、并发管理、堆栈管理等其它Go语言的关键特性。这个所谓的runtime可以类比于c语言的标准库(libc)</p>

<p>但很重要的一点是要理解，Go的这个所谓的runtime并没有带虚拟机（比如，像Java一样）。所以，尽管runtime这个词一般用来指某些语言带的虚拟机实现，但在Golang的语境下，它仅仅指代标准库。</p>

<blockquote><p>Why does Go not have generic types? / 为什么没有泛型？</p></blockquote>

<p>泛型在将来的某个时候可能会被加上。但我们并不着急加它，虽然我们理解有些程序员可能不认同。</p>

<p>Go的目的是用于服务端程序开发，它的设计的目标在于增强程序的可扩展性、并发能力以及代码的可读性。泛型对于这些设计目标并没有多大帮助，所以为了实现的简单，当时并没有支持泛型。</p>

<p>现在Go语言已经越来越成熟了，可以考虑加上泛型的支持，但是对于泛型我还是想多说几句。</p>

<p>泛型很方便，但是它大大增加了语言的类型系统(type system)和运行时的实现复杂度，我们还没有找到一种能很好平衡两者的设计方案。目前大家先用<code>interface{}</code>来顶一顶吧。</p>

<blockquote><p>Why does Go not have exceptions? / 为什么没有异常？</p></blockquote>

<p>我们认为把exception耦合进正常逻辑中（比如其它语言中常见的<code>try-catch-finally</code>惯用法）会导致混乱的代码。使用exception的方式也倾向于鼓励程序员把很多普通的错误（比如无法打开文件）当成异常来处理。</p>

<p>对于普通的错误，需要在函数调用后立即处理。</p>

<p>Go也提供了一组内置函数用来处理真正的异常情况（panic和recovery）。</p>

<p>linuxfish注：我个人理解，关键是要区分普通错误和真正的异常。</p>

<h3>Types</h3>

<blockquote><p>Why does Go not support overloading of methods and operators? / 为何不支持方法和运算符重载？</p></blockquote>

<p>在其它语言上的经验告诉我们，搞一堆名字一样，签名不同的方法，只是偶尔有点用处，更多的会带来理解上的混乱。相反，不支持重载大大简化了Go语言的类型系统的设计。</p>

<p>至于运算符重载，在我们看来它可以带来一定的便利性，但并非必须。按照我们的惯例，宁缺毋滥。</p>

<blockquote><p>Why doesn&rsquo;t Go have &ldquo;implements&rdquo; declarations? / 为何不用<code>implements</code>显示声明？</p></blockquote>

<p>一个Go的类型如果实现了某个interface的方法，那它就隐式的“声明”为这个interface类型了（即可以当成这个interface类型来用了）。</p>

<p>好处是实现了接口和实现的真正解耦，允许在不更改现有代码的前提下定义和使用接口。</p>

<blockquote><p>Can I convert a []T to an []interface{}?</p></blockquote>

<p>不能直接转，因为它们在内存中的表示并不一样，需要这样做：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">t</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">int</span><span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">}</span>
</span><span class='line'><span class="nx">s</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kd">interface</span><span class="p">{},</span> <span class="nb">len</span><span class="p">(</span><span class="nx">t</span><span class="p">))</span>
</span><span class='line'><span class="k">for</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">t</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">s</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="p">=</span> <span class="nx">v</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>一个更通用但也更慢的函数：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">InterfaceSlice</span><span class="p">(</span><span class="nx">slice</span> <span class="kd">interface</span><span class="p">{})</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{}</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">s</span> <span class="o">:=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">slice</span><span class="p">)</span>
</span><span class='line'>    <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Kind</span><span class="p">()</span> <span class="o">!=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Slice</span> <span class="p">{</span>
</span><span class='line'>        <span class="nb">panic</span><span class="p">(</span><span class="s">&quot;InterfaceSlice() given a non-slice type&quot;</span><span class="p">)</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">ret</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kd">interface</span><span class="p">{},</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Len</span><span class="p">())</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">for</span> <span class="nx">i</span><span class="o">:=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">i</span><span class="p">&lt;</span><span class="nx">s</span><span class="p">.</span><span class="nx">Len</span><span class="p">();</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">ret</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Index</span><span class="p">(</span><span class="nx">i</span><span class="p">).</span><span class="nx">Interface</span><span class="p">()</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nx">ret</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>参考：<a href="https://stackoverflow.com/questions/12753805/type-converting-slices-of-interfaces-in-go">Type converting slices of interfaces in go</a></p>

<blockquote><p>Can I convert []T1 to []T2 if T1 and T2 have the same underlying type?</p></blockquote>

<p>不可以，只能一个一个转。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">type</span> <span class="nx">T1</span> <span class="kt">int</span>
</span><span class='line'><span class="kd">type</span> <span class="nx">T2</span> <span class="kt">int</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">t1</span> <span class="nx">T1</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">x</span> <span class="p">=</span> <span class="nx">T2</span><span class="p">(</span><span class="nx">t1</span><span class="p">)</span> <span class="c1">// OK</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">st1</span> <span class="p">[]</span><span class="nx">T1</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">sx</span> <span class="p">=</span> <span class="p">([]</span><span class="nx">T2</span><span class="p">)(</span><span class="nx">st1</span><span class="p">)</span> <span class="c1">// NOT OK</span>
</span></code></pre></td></tr></table></div></figure>


<blockquote><p>Why is my nil error value not equal to nil?</p></blockquote>

<p>首先，interface类型的底层实现是一个二元组（一个类型T和一个值V），V就是具体的值，T就是这个值的类型。比如，我们存了一个int 3在接口里，那么用字符简单的表示，这个接口值现在就等于<code>(T=int, V=3)</code>。</p>

<p>一个接口值当且仅当V和T都未设置时（T=nil, V未设置）才为nil。特别地，一个值为nil的接口的底层T永远都是nil。那么，如果把一个值为nil类型为<code>*int</code>存入一个接口中，此时这个接口值的底层表示为<code>(T=*int, V=nil)</code>，按照定义，这个接口值不等于nil。</p>

<p>但这跟我们的日常认知相悖，明明是个空指针嘛！这种情况在从函数中返回error时更容易使人困惑：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">returnsError</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">p</span> <span class="o">*</span><span class="nx">MyError</span> <span class="p">=</span> <span class="kc">nil</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">bad</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="nx">p</span> <span class="p">=</span> <span class="nx">ErrBad</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="nx">p</span> <span class="c1">// Will always return a non-nil error.</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>上面的写法的错误之处在于，不管有没有错误发生，函数返回的error永远都是non-nil的，所以当无错误发生时，应当显式地返回nil：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">returnsError</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">if</span> <span class="nx">bad</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">ErrBad</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="k">return</span> <span class="kc">nil</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>Values</h3>

<blockquote><p>Why does Go not provide implicit numeric conversions? / 为何不提供隐式数字类型转换？</p></blockquote>

<p>c语言中的自动数值类型转换带来的那点便利性跟它引起的困惑相比根本不值得一提。表达式的结果什么时候是无符号的？它的值有多大？是不是溢出了？结果是不是可移植的（执行结果跟程序所在的机器无关）？考虑到移植性的问题，我们要求在进行类型转换时必须进行显式的转换，尽管有些麻烦。</p>

<p>一个相关的细节是，就算<code>int</code>实质上是一个<code>int64</code>（当在64位机器上时），<code>int</code>和<code>int64</code>也是两个不同的类型。如果你真的关心一个应该用多少位来表示一个整数，我们鼓励你显式地声明它（用int32或者int64，而不是int）。</p>

<blockquote><p>How do constants work in Go?</p></blockquote>

<p>尽管Go要求不同的数值类型必须进行显式的转换，它对常数值的要求比较灵活。因为常数值默认是无类型的（untyped），所以不必这么写：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">sqrt2</span> <span class="o">:=</span> <span class="nx">math</span><span class="p">.</span><span class="nx">Sqrt</span><span class="p">(</span><span class="nb">float64</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>你可以直接写：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">sqrt2</span> <span class="o">:=</span> <span class="nx">math</span><span class="p">.</span><span class="nx">Sqrt</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>想了解Go中的常数值的具体实现的可以参考<a href="https://blog.golang.org/constants">这篇Blog</a></p>

<blockquote><p>Should I define methods on values or pointers? / 方法定义在值上还是指针上？</p></blockquote>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">MyStruct</span><span class="p">)</span> <span class="nx">pointerMethod</span><span class="p">()</span> <span class="p">{</span> <span class="p">}</span> <span class="c1">// method on pointer</span>
</span><span class='line'><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="nx">MyStruct</span><span class="p">)</span>  <span class="nx">valueMethod</span><span class="p">()</span>   <span class="p">{</span> <span class="p">}</span> <span class="c1">// method on value</span>
</span></code></pre></td></tr></table></div></figure>


<p>接收者是要定义成值还是指针本质上跟函数的参数是定义成值还是指针是一个问题。</p>

<p>我们在回答这个问题时通常有以下考虑：</p>

<ul>
<li>方法是否需要修改这个接收者？如果需要修改，那没有什么纠结的，必须用指针（slice 和 map 本身就是引用类型，需要单独讨论）。</li>
<li>效率上的考虑。如果接收者是一个大结构体，那么很显然用指针更合适</li>
<li>一致性。如果有些方法必须用指针做为接收者，那么为了使用上的一致，其它的方法也应该用指针</li>
<li>除非必须，一般情况下使用值就可以了</li>
</ul>


<blockquote><p>What&rsquo;s the difference between new and make? / new 和 make 的区别？</p></blockquote>

<p>简而言之，new 用来分配内存，而 make 用来初始化slice，map 和 channel。</p>

<p>具体解释可参考 Effective Go 中的<a href="https://golang.org/doc/effective_go.html#allocation_new">相关章节</a></p>

<h3>Concurrency</h3>

<blockquote><p>Why doesn&rsquo;t my program run faster with more CPUs?</p></blockquote>

<p>一个程序是否会因为有更多的CPU资源而运行更快取决于这个程序所要解决的问题。Go语言提供了基本的并发原语，比如 goroutine 和 channel，但只有当要解决的问题存在并行执行的可能性时才会发挥它们的作用。本质上是串行的问题，仅仅通过增加更多的CPU资源无法获得更高的执行效率。</p>

<h3>Functions and Methods</h3>

<blockquote><p>What happens with closures running as goroutines? / 闭包的坑</p></blockquote>

<p>在并发中使用闭包时，可能会出现一些令人困惑的现象，考虑以下代码：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">done</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">bool</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>    <span class="nx">values</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&quot;a&quot;</span><span class="p">,</span> <span class="s">&quot;b&quot;</span><span class="p">,</span> <span class="s">&quot;c&quot;</span><span class="p">}</span>
</span><span class='line'>    <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">values</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span>
</span><span class='line'>            <span class="nx">done</span> <span class="o">&lt;-</span> <span class="kc">true</span>
</span><span class='line'>        <span class="p">}()</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// wait for all goroutines to complete before exiting</span>
</span><span class='line'>    <span class="k">for</span> <span class="nx">_</span> <span class="p">=</span> <span class="k">range</span> <span class="nx">values</span> <span class="p">{</span>
</span><span class='line'>        <span class="o">&lt;-</span><span class="nx">done</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>我们可能错误的认为输出结果应该是<code>a,b,c</code>，但实际上你看到的结果可能是<code>c,c,c</code>。这是因为所有的闭包都共享了一个变量v，而v在循环中被不断的更新，当闭包中的程序真正在另一个goroutine中运行时，v的值已经变了。使用<code>go vet</code>可以在运行程序前提前检测这种情况。</p>

<p>针对这个“坑”，有两个解决方案：</p>

<p>要么，把v显式传入闭包中</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">values</span> <span class="p">{</span>
</span><span class='line'>    <span class="k">go</span> <span class="kd">func</span><span class="p">(</span><span class="nx">u</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">u</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">done</span> <span class="o">&lt;-</span> <span class="kc">true</span>
</span><span class='line'>    <span class="p">}(</span><span class="nx">v</span><span class="p">)</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>或者这样写：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">values</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">v</span> <span class="o">:=</span> <span class="nx">v</span> <span class="c1">// create a new &#39;v&#39;.</span>
</span><span class='line'>    <span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span>
</span><span class='line'>        <span class="nx">done</span> <span class="o">&lt;-</span> <span class="kc">true</span>
</span><span class='line'>    <span class="p">}()</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>Control flow</h3>

<blockquote><p>Why does Go not have the ?: operator? / 为何没有<code>?:</code>操作符？</p></blockquote>

<p>可以使用以下代码实现类似的效果：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="k">if</span> <span class="nx">expr</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">n</span> <span class="p">=</span> <span class="nx">trueVal</span>
</span><span class='line'><span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">n</span> <span class="p">=</span> <span class="nx">falseVal</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Go中不存在这个三元操作符的原因是Go语言的设计者认为这个三元操作符经常被用来写出晦涩难懂的表达式。<code>if-else</code>的模式，虽然写起来更长一些，但毫无疑问表达的更清楚。一门语言只需要一种条件控制结构就够了。</p>

<h3>Packages and Testing</h3>

<blockquote><p>Where is my favorite helper function for testing?</p></blockquote>

<p>使用Go标准库中的测试包很容易写单元测试，但这个测试包缺乏一些其它测试框架中常用的功能，比如断言。之前的章节中已经解释了为啥Go不支持断言，这个解释同样适用于写测试。在进行单元测试时，正确的错误处理方式是在某个测试出错后，允许其它的测试继续进行，这样进行测试的人可以从全局看到是什么地方出了问题。举个例子，现在正在测试 isPrime 这个函数，那么与其只报告了isPrime对于2这个输入的结果是错的，然后就停止测试了，直接报告 isPrime 对于输入（2，3，5，7）的结果都是错的明显更有用处。运行测试代码的人可能对被测试的代码并不熟悉，那么在写测试代码时，好好写一个错误提示信息就很有必要了，能帮助后面测试出现错误定位问题时省不少力气。</p>

<p>还有一点是测试框架都喜欢定义自己的一套用来表示条件、控制语句和打印等固定模式的DSL，但问题是Go本身就能实现这些功能，为什么还要重复造轮子呢？可以少学一门语言，何乐而不为呢？</p>

<p>如果你觉得用Go的测试包来写测试太啰嗦了，会有很多重复的代码，那么你可以考虑用驱动表的方式来写测试，这样的话，重复的代码会大大减少。Go标准库中有大量的测试代码可供你参考。</p>

<h3>Implementation</h3>

<blockquote><p>Why is my trivial program such a large binary? / 为啥那么点代码就编译出这么大一个二进制文件？</p></blockquote>

<p>gc 工具链中的链接器(linker)默认创建静态链接的二进制文件。 因此所有的 Go 二进制文件包含了Go的运行时，同时还有执行动态类型检测、反射甚至崩溃时的堆栈信息的运行时</p>

<p>一个简单的 hello world 版 c 程序，用 gcc 在 linux 上静态编译后，体积大约是750KB。同样的 Go 程序，体积大概有几M，但功能也更强大，因为它有 run time 支持，还有动态类型检查和 debug 信息。</p>

<p>Go程序在用 gc 编译时可以带上<code>-ldflags=-w</code>参数来关闭 debug 信息的生成，这不会影响程序的正常功能，但会很大程度地减少程序的体积。</p>

<blockquote><p>Can I stop these complaints about my unused variable/import? / 能不能别老提示我未用的变量和引入库？</p></blockquote>

<p>代码中存在未使用的变量可能意味着潜在的 bug，而未使用的引用库只会增大编译时间，而且随着代码量的增长，这个问题会越来越严重。基于以上原因 Go 拒绝编译存在未使用变量和未使用的引用库的代码，短期来看，这个决定带来了很多麻烦，但从长远来看，它利于提升程序的编译速度和代码的整洁性。</p>

<p>当然了，在写代码的时候，创建一堆临时变量是常见的，每次都必须要先处理一下这些用不上的临时变量才能编译确实是挺烦的。</p>

<p>有人提议加一个编译器的参数可以关闭这个傻X的检查，或者至少编译能通过，比如只是发个警告信息提示用户。但我们没有加这个参数，一是因为我们认为编译器的参数不应该影响语言的语义；二是 Go 编译器没有警告，它只报告错误。</p>

<p>Go编译器没有警告，原因有二：一是，如果一个问题需要报警提示，那它就应该被改正（相反，如果它不值得被修正，那也根本无需发警告提示）；第二，有时候太多警告反而会掩盖了真实的问题。</p>

<p>使用空白标识符<code>_</code>可以一定程度的缓解这个问题。</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="kn">import</span> <span class="s">&quot;unused&quot;</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// This declaration marks the import as used by referencing an</span>
</span><span class='line'><span class="c1">// item from the package.</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">_</span> <span class="p">=</span> <span class="nx">unused</span><span class="p">.</span><span class="nx">Item</span>  <span class="c1">// TODO: Delete before committing!</span>
</span><span class='line'>
</span><span class='line'><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">debugData</span> <span class="o">:=</span> <span class="nx">debug</span><span class="p">.</span><span class="nx">Profile</span><span class="p">()</span>
</span><span class='line'>    <span class="nx">_</span> <span class="p">=</span> <span class="nx">debugData</span> <span class="c1">// Used only during debugging.</span>
</span><span class='line'>    <span class="o">...</span><span class="p">.</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>当然了，现在大部分Go程序员都知道有这个神器，goimports，它能分析你的源文件，自动清理掉未用到的变量，在你保存文件的时候自动重写源文件，大部分的编辑器都支持它。</p>

<h3>Changes from C</h3>

<blockquote><p>Why is there no pointer arithmetic? / 为啥不支持指针运算？</p></blockquote>

<p>安全。不支持指针运算能避免很多诡异的问题。编译器和硬件技术发展到现在，用数组索引和用指针来遍历一个数组，在效率上已经没有什么区别。另外，不支持指针运算使得垃圾回收也更好实现了。</p>

<blockquote><p>Why are ++ and &mdash; statements and not expressions? And why postfix, not prefix?</p></blockquote>

<p>不支持指针运算的话，那么对<code>++</code>和<code>--</code>的需求就没那么大了。通过把<code>++</code>和<code>--</code>限制为 statement，而不是表达式(expression)，我们可以避免类似以下写法的出现：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='go'><span class='line'><span class="nx">f</span><span class="p">(</span><span class="nx">i</span><span class="o">++</span><span class="p">)</span>
</span><span class='line'><span class="nx">p</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="p">=</span> <span class="nx">q</span><span class="p">[</span><span class="o">++</span><span class="nx">i</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[如何处理数据库大结果集?]]></title>
    <link href="http://cs50Mu.github.io/blog/2017/11/26/big-resultsets-handling-in-django/"/>
    <updated>2017-11-26T23:57:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2017/11/26/big-resultsets-handling-in-django</id>
    <content type="html"><![CDATA[<p>问题的起因是想知道在Django ORM中如何处理大数据集的返回，比如怎么避免进程由于内存占用过多被kill掉。由于数据库使用的是MySQL，讨论是从MySQL开始的。</p>

<p>MySQL协议是半双工的，同一时间只能有一方在说话，一旦服务器开始返回数据，客户端能做的只能是把数据接收完，不存在让服务器停止这一说。。（因为还没轮到它说嘛。。）所以，写sql的时候注意加limit是很重要的！</p>

<p>那么对于一次sql查询的返回，MySQL客户端有两个选择：</p>

<p>1、MySQL客户端默认会一次性先把服务器返回的数据先缓存起来，再给它的客户去用。这样做的好处是，能尽快解放数据库相关线程，让它们去做更重要的事，比如服务其它请求（因为服务器通常需要等所有数据都发送完后才释放这条查询相关的资源的），坏处是如果返回的结果很大，客户端需要花费很多时间和内存来接收它，更气人的是，在这期间，做为客户端的用户，你啥也干不了，只能等。</p>

<p>2、MySQL客户端还有一个选择，就是不自己先缓存啦，一边从server接收一边就返回给它的用户了~这样做的好处是，对于MySQL客户端的用户来说，它看起来反应更快了（因为没有先缓存所有数据集），而且貌似还便于做内存占用的优化？比如一边接收、处理，一边删除已处理过的数据，使得内存占用始终保持在一个很小的数量。坏处是有一个处理很慢的用户可能会拖垮数据库，并且在接收完之前客户端不能做任何其它的事情！这其实是一个Unbuffered Cursor，在MySQL客户端实现里叫SSCursor，即server side cursor，但其实它不是真正的server side cursor</p>

<p>Django ORM的queryset在被使用的时候会触发对应sql查询被执行，像上面说的，默认情况下结果集会被客户端先缓存，这已经是一个内存占用，然后会被Django转成对应的model instance，这又是一个内存占用，如果内存没有被及时回收，这其实是一份数据的双倍内存占用。</p>

<p>queryset的一个方法iterator能实现的一个优化是，将数据集转成model instance的过程改为generator模式，减少第二步的内存占用（注意：这只是针对MySQL来说的）</p>

<p>对于会返回大数据集的查询，一个处理办法是拆分使用limit来多次接收。每次接收一定量的数据（比如1000），内存回收后再接收下一个1000，对应的Django实现：</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">gc</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">queryset_iterator</span><span class="p">(</span><span class="n">queryset</span><span class="p">,</span> <span class="n">chunksize</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
</span><span class='line'>    <span class="sd">&#39;&#39;&#39;&#39;&#39;</span>
</span><span class='line'><span class="sd">    Iterate over a Django Queryset ordered by the primary key</span>
</span><span class='line'>
</span><span class='line'><span class="sd">    This method loads a maximum of chunksize (default: 1000) rows in it&#39;s</span>
</span><span class='line'><span class="sd">    memory at the same time while django normally would load all rows in it&#39;s</span>
</span><span class='line'><span class="sd">    memory. Using the iterator() method only causes it to not preload all the</span>
</span><span class='line'><span class="sd">    classes.</span>
</span><span class='line'>
</span><span class='line'><span class="sd">    Note that the implementation of the iterator does not support ordered query sets.</span>
</span><span class='line'><span class="sd">    &#39;&#39;&#39;</span>
</span><span class='line'>    <span class="n">pk</span> <span class="o">=</span> <span class="mi">0</span>
</span><span class='line'>    <span class="n">last_pk</span> <span class="o">=</span> <span class="n">queryset</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s">&#39;-pk&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">pk</span>
</span><span class='line'>    <span class="n">queryset</span> <span class="o">=</span> <span class="n">queryset</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s">&#39;pk&#39;</span><span class="p">)</span>
</span><span class='line'>    <span class="k">while</span> <span class="n">pk</span> <span class="o">&lt;</span> <span class="n">last_pk</span><span class="p">:</span>
</span><span class='line'>        <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">queryset</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">pk__gt</span><span class="o">=</span><span class="n">pk</span><span class="p">)[:</span><span class="n">chunksize</span><span class="p">]:</span>
</span><span class='line'>            <span class="n">pk</span> <span class="o">=</span> <span class="n">row</span><span class="o">.</span><span class="n">pk</span>
</span><span class='line'>            <span class="k">yield</span> <span class="n">row</span>
</span><span class='line'>        <span class="n">gc</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p>参考：</p>

<ul>
<li><a href="https://stackoverflow.com/questions/4856882/limiting-memory-use-in-a-large-django-queryset?rq=1">https://stackoverflow.com/questions/4856882/limiting-memory-use-in-a-large-django-queryset?rq=1</a></li>
<li><a href="https://stackoverflow.com/questions/14144408/memory-efficient-constant-and-speed-optimized-iteration-over-a-large-table-in?noredirect=1&amp;lq=1">https://stackoverflow.com/questions/14144408/memory-efficient-constant-and-speed-optimized-iteration-over-a-large-table-in?noredirect=1&amp;lq=1</a></li>
<li><a href="https://djangosnippets.org/snippets/1949/">https://djangosnippets.org/snippets/1949/</a></li>
<li><a href="https://docs.djangoproject.com/en/1.11/ref/models/querysets/#iterator">https://docs.djangoproject.com/en/1.11/ref/models/querysets/#iterator</a></li>
</ul>


<p>下面讲一下真正的server side cursor，postgresql支持真正意义上的server side cursor，可以给一个查询指定一个cursor，然后在这个cursor上操作，fetch多行、move(移动）cursor、更新当前cursor所在的记录等等，可参考<a href="https://www.postgresql.org/docs/9.2/static/plpgsql-cursors.html">文档</a>。看MySQL<a href="https://dev.MySQL.com/doc/refman/5.7/en/cursors.html">官方文档</a>，它也支持一个很简陋的server side cursor，而且只能在存储过程里用。</p>

<p>顺便，google到一个对比Client-Side Cursors和Server-Side Cursors区别的文档，竟然来自微软。。</p>

<p>Client-Side Cursors</p>

<p>With a non-keyset client-side cursor, the server sends the entire result set across the network to the client machine. The client machine provides and manages the temporary resources needed by the cursor and result set. The client-side application can browse through the entire result set to determine which rows it requires.</p>

<p>Static and keyset-driven client-side cursors may place a significant load on your workstation if they include too many rows. While all of the cursor libraries are capable of building cursors with thousands of rows, applications designed to fetch such large rowsets may perform poorly. There are exceptions, of course. For some applications, a large client-side cursor may be perfectly appropriate and performance may not be an issue.</p>

<p>One obvious benefit of the client-side cursor is quick response. After the result set has been downloaded to the client machine, browsing through the rows is very fast. Your application is generally more scalable with client-side cursors because the cursor&rsquo;s resource requirements are placed on each separate client and not on the server.</p>

<p>Server-Side Cursors</p>

<p>With a server-side cursor, the server manages the result set using resources provided by the server machine. The server-side cursor returns only the requested data over the network. This type of cursor can sometimes provide better performance than the client-side cursor, especially in situations where excessive network traffic is a problem.</p>

<p>Server-side cursors also permit more than one operation on the connection. That is, once you create the cursor, you can use the same connection to make changes to the rows — without having to establish an additional connection to handle the underlying update queries.</p>

<p>However, it&rsquo;s important to point out that a server-side cursor is — at least temporarily — consuming precious server resources for every active client. You must plan accordingly to ensure that your server hardware is capable of managing all of the server-side cursors requested by active clients. Also, a server-side cursor can be slow because it provides only single row access — there is no batch cursor available.</p>

<p>Server-side cursors are useful when inserting, updating, or deleting records. With server-side cursors, you can have multiple active statements on the same connection. With SQL Server, you can have pending results in multiple statement handles.</p>

<p>参考：</p>

<ul>
<li><a href="https://msdn.microsoft.com/en-us/library/aa266531(v=vs.60">Client-Side Cursors Versus Server-Side Cursors</a>.aspx)</li>
<li><a href="http://thebuild.com/blog/2010/12/13/very-large-result-sets-in-django-using-postgresql/">Very Large Result Sets in Django using PostgreSQL</a></li>
<li><a href="https://www.postgresql.org/docs/9.2/static/plpgsql-cursors.html">postgresql cursors</a></li>
<li><a href="http://techualization.blogspot.com/2011/12/retrieving-million-of-rows-from-MySQL.html">Retrieving million of rows from MySQL</a></li>
<li><a href="https://dev.MySQL.com/doc/refman/5.7/en/cursors.html">MySQL cursors</a></li>
<li><a href="https://code.djangoproject.com/ticket/16614#no1">Django中对server side cursor的支持</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[chunked encoding and wsgi]]></title>
    <link href="http://cs50Mu.github.io/blog/2017/10/15/chunked-encoding-and-wsgi/"/>
    <updated>2017-10-15T15:58:00+08:00</updated>
    <id>http://cs50Mu.github.io/blog/2017/10/15/chunked-encoding-and-wsgi</id>
    <content type="html"><![CDATA[<p>记录一次问题定位过程。</p>

<p>一天，接到业务组的同学反馈：“你们的某个服务在仿真环境不可用了”。看到报错信息后我第一反应就是，是他们传递的参数有误，因为：1. 近期代码没有改动，只是由ECS
部署改为了Docker部署；2. 报错信息提示post的json数据有问题。</p>

<p>然而，跟业务组同学反复确认后发现，他们post的数据并没有错。。进一步定位后发现，原来不是post参数有误，而是应用里读取到的post data为空了！</p>

<p>跑到运维同学那里，进入容器环境用tcpdump抓了下包，发现进入应用框架(Django)之前post数据还是在的。。</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>tcpdump -l -s 0 -w - dst port 5000 | strings</span></code></pre></td></tr></table></div></figure>


<p>抓包数据如下：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>POST /api/purchases/ HTTP/1.1
</span><span class='line'>Host: jccoin.dev.igetget.com
</span><span class='line'>User-Agent: Go-http-client/1.1
</span><span class='line'>Transfer-Encoding: chunked
</span><span class='line'>Content-Type: application/json
</span><span class='line'>Accept-Encoding: gzip
</span><span class='line'>X-Forwarded-For: 10.30.47.54
</span><span class='line'>Connection: close
</span><span class='line'>{"purchase_no":"b7eb4vd86rijdi8i5sa0","category":2,"sys_code":"IGET","buyer_id":"21075408","device_type":"ANDROID","product_category":"53","product_id":"1","product_name":"
</span><span class='line'>","coins_amount":335,"tag":"","sign":"ffc88ffc0b0aaf191d11b0909575f40c"}</span></code></pre></td></tr></table></div></figure>


<p>但是，这怎么可能？！</p>

<p>切换回ECS环境后接口恢复正常了，于是在ECS环境重新抓包如下：</p>

<p>ECS环境上应用框架(Django)前有一个nginx，进入nginx前的包数据如下：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>POST /api/purchases/ HTTP/1.1
</span><span class='line'>Host: jccoin.dev.igetget.com
</span><span class='line'>User-Agent: Go-http-client/1.1
</span><span class='line'>Transfer-Encoding: chunked
</span><span class='line'>Content-Type: application/json
</span><span class='line'>Accept-Encoding: gzip
</span><span class='line'>{"purchase_no":"b7ep9jt86rik99q3q6eg","category":2,"sys_code":"IGET","buyer_id":"645563","device_type":"ANDROID","product_category":"55","product_id":"2","product_name":"30
</span><span class='line'>","coins_amount":48,"tag":"30","sign":"68fff62307beec83399933c5d2bd278d"}</span></code></pre></td></tr></table></div></figure>


<p>从nginx出来，进入应用框架(Django)之前的包数据如下：</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>POST /api/purchases/ HTTP/1.0
</span><span class='line'>Host: jccoin.dev.igetget.com
</span><span class='line'>X-Real-IP: 10.30.47.54
</span><span class='line'>X-Forwarded-For: 10.30.47.54
</span><span class='line'>Connection: close
</span><span class='line'>Content-Length: 263
</span><span class='line'>User-Agent: Go-http-client/1.1
</span><span class='line'>Content-Type: application/json
</span><span class='line'>Accept-Encoding: gzip
</span><span class='line'>{"purchase_no":"b7ep9jt86rik99q3q6eg","category":2,"sys_code":"IGET","buyer_id":"645563","device_type":"ANDROID","product_category":"55","product_id":"2","product_name":"30
</span><span class='line'>","coins_amount":48,"tag":"30","sign":"68fff62307beec83399933c5d2bd278d"}</span></code></pre></td></tr></table></div></figure>


<p>可以看到，经过nginx后数据由chunked变成了content-length。</p>

<p>从上面的情况猜测应用框架对chunked encoding支持有问题，一番google后发现果然如此:</p>

<ul>
<li><a href="https://stackoverflow.com/questions/12091067/handling-http-chunked-encoding-with-django">https://stackoverflow.com/questions/12091067/handling-http-chunked-encoding-with-django</a></li>
<li><a href="https://github.com/pallets/flask/issues/2229">https://github.com/pallets/flask/issues/2229</a></li>
<li><a href="http://mathslinux.org/?p=564">http://mathslinux.org/?p=564</a></li>
<li><p><a href="https://twitter.com/davidism/status/888426616602230784">https://twitter.com/davidism/status/888426616602230784</a></p></li>
<li><p><a href="http://lucumr.pocoo.org/2011/7/27/the-pluggable-pipedream/">http://lucumr.pocoo.org/2011/7/27/the-pluggable-pipedream/</a></p></li>
<li><a href="https://www.python.org/dev/peps/pep-0333/#handling-the-content-length-header">https://www.python.org/dev/peps/pep-0333/#handling-the-content-length-header</a></li>
<li><a href="http://rhodesmill.org/brandon/2013/chunked-wsgi/">http://rhodesmill.org/brandon/2013/chunked-wsgi/</a></li>
</ul>


<p>所以这么看来，Django这一类框架在前面挂一个nginx是必须的。</p>
]]></content>
  </entry>
  
</feed>