Scaling of kernel functions

[DOSull]

Not really an issue, as I don't use mgwr (or haven't yet), more a question.

Are the constant scale factors applied to the kernel functions in _kernel_funcs(self, zs) here correct?

Perhaps more relevant, in the GWR context are they even necessary?

I ask because there is an odd mixture of constants applied to the kernels in some cases and not applied in others.

In density estimation applications of spatial kernels, volume preservation under the density surface is important. If this matters in the GWR case, then rather obviously the triangular function is not volume preserving as written given that the volume of a cone is $\pi r^2h/3$ for a cone of height $h$ and base radius $r$. This means that a volume preserving triangular kernel of bandwidth $w$ would have height (i.e. constant multiplier) $3/\pi w^2$. I assume since that's not applied (similarly the Gaussian and quartic kernels as written are not volume preserving, and none of the others are either, given that none of them are parameterised by a bandwidth.

They're not really kernels, which usually implies estimation of a PDF, so much as spatial interaction functions.

Anyway... if volume preservation doesn't matter then that means there's no point in applying any constant scaling factors to any of them so that the (3. / 4) applied to the quadratic and the (15. / 16) applied to the quartic can safely be dropped, and some infinitesimal amount of time saved!

[TaylorOshan]

Hi @DOSull, volume preservation doesn’t matter in the sense of reproducing 'canonical' results, however, I'm not sure how it would or would not impact GWR estimation. I suspect not, or at least not significantly.

They're not really kernels, which usually implies estimation of a PDF, so much as spatial interaction functions.

I think that’s a fair point.

I didn’t follow the comment about none of them being parameterized by a bandwidth. GWR always uses a bandwidth to determine the distances (zs) used in those functions - I guess that's fundamentally different than the bandwidth, w, you mention.

FWIW, the triangular, quadratic, and quartic functions don’t see much play in GWR. The set of functions was ported from the kernel module of the weights package and then modified to remove truncations (i.e., arbitrary removal of near-zero interactions) that existed for memory optimization.

[ljwolf]

I think @DOSull means that, without a (1/(sqrt(2 * pi)*bw)) scaling factor, the Gaussian kernel we estimate is only proportional to the "correct" one that has that scaling factor.

Imho we should keep everything volume preserving. I'm not aware of this being studied in GWR, but I bet it's been studied in GAMs?

Reasoning through it, the constants should cancel out within the WLS estimator, no? However, we would get in trouble for many other estimators or statistics, so the kernel should be fully specified.

[DOSull]

My (limited) understanding of GWR would be that all these scaling factors will cancel out since it's all relative anyway.

This was just something I came across along the way and as a good FOSS citizen thought I should mention. I came across it in the context of a round of updating of the CSR model in our NetLogo model zoo where I realised that our quartic kernel method and diffusion smoothing method were yielding different total volumes under the surface, because we were certainly using the wrong scaling factor for the quartic function option!

That led me to the books, one of which was Fotheringham et al. 2002 on GWR, and from there to here. There are surprisingly few explicit statements out there of the $k$ scaling factor on kernel functions anywhere in the literature, and many of them restrict themselves to the scaling factor in the 1D case... which it need hardly be said is irrelevant here... I also have no idea where the reference in the code # functions follow Anselin and Rey (2010) table 5.4 is pointing to as there is no Table 5.4 in either of the obvious possible references (Perspectives on Spatial Data Analysis has no Table 5.4 and only one reference to kernels; and Serge and Luc's chapter in Handbook of Applied Spatial Analysis about PySAL is Chapter A10).

Returning to mgwr it isn't so much the correctness (or not) in terms of volume preservation just the inconsistency that struck me. As noted in my comment if volume preservation isn't a concern then there is no point at all in the various scaling factors for some of the kernels and you might as well get rid of them for whatever minor performance improvement that might yield.

Alternatively, as @ljwolf points out, the kernel should be fully specified on some principled basis if only so that competing implementations are consistent on such things and don't inadvertently yield different results for obscure reasons related to this kind of minor difference!

[geoglrb]

Hello! This is an exciting Issue, at least to me. Whether the normalization of those functions matters seems (to me) to be a matter of how you've coded the rest of things, which I think @DOSull and I plead guilty to not having explored beyond the code directly in question above, either by testing or reading/reasoning.

I am selfishly interested in the answer as it has some bearing on something I'm interested in seeing, feature-wise. I wonder whether I could get a spatial statistician cleverer than I am to comment on what would be needed to implement a generalized data-driven kernel option--an analogue to a W matrix for GWR. Both @TaylorOshan and @ljwolf seem to be well-positioned to think on this, given https://doi.org/10.1080/13658816.2022.2034829 and many others! ;-)

Also https://doi.org/10.1111/tgis.12020

[ljwolf]

Just to update: we've discussed this in our development meeting today---broadly speaking, there is no clear consensus on whether "volume-preserving" (a.k.a "proper") kernels should be default, because of the following confusion.

Kernels in a statistical sense are components of a probability density function. This is the sense in which kernels are used in spreg, mgwr, and the new gwlearn. These usually need to be volume-preserving/proper. If they do not integrate to 1, then they can cause statistical estimates be inflated/deflated proportional to some arbitrary constant.

For many users, we offer kernels to represent a spatial affinity function in situations where they intend to model distance decay. This is the sense that they are used in libpysal and the new gwlearn. Such a spatial affinity would vary between "1" (completely similar at distance=0) to "0" (completely dissimilar at some large distance, possibly the bandwidth). In this sense, kernels ought to have K(0)=1, since observations that are co-located should be the most similar in the kernel.

To achieve K(u=0)=1 in all kernels except triangular, we would need to omit the scaling constants---the parabolic kernel, for instance, is a proper kernel with K(u=0)=3/4. To obtain K(u=0)=1 with parabolic decay, we'd define a parabolic decay function D(u)=(1-u^2) and set the kernel K(u)=D(u)*3/4. Statistical packages could choose whichever form to use. Most would tend to use the kernel form, while libpysal might make the decay form default, given spatial weights tend to represent the second sense.

@DOSull, @geoglrb, any perspective?
(@martinfleis, @sjsrey: this reflect your recollection of the discussion?)

[martinfleis]

Thanks @ljwolf for a nice overview.

Just one thing to add, regarding the original question on whether it is correct or if it even matters in the context of GWR. In (M)GWR and derived models, we use kernel function to get the distance decay, which captures the importance of the observation for each focal point. In that sense, it is more natural to represent weights with K(0)=1. However, in practice it does not matter. The code snippet below simulates what happens in gwlearn (which matches mgwr for GWR) and as you can see, given the relative proportion of weights between the samples remains the same no matter the scaling, the outcome is exactly the same.

... probability density function. This is the sense in which kernels are used in spreg, mgwr

@ljwolf this is not correct for mgwr, it implements the gaussian, bisquare and exponential kernels as K(0)=1. The rest is a copy&paste from libpysal which is not documented and shall not be used imho (triangular can yield negative weights).

mgwr/mgwr/kernels.py

Lines 76 to 80 in aafdf6a

	return np.exp(-0.5 * (zs)**2)
	elif self.function == 'bisquare':
	return (1 - (zs)**2)**2
	elif self.function == 'exponential':
	return np.exp(-zs)

In [1]: import numpy as np
   ...: from sklearn.datasets import load_diabetes
   ...: from sklearn.linear_model import LinearRegression
   ...: 
   ...: X, y = load_diabetes(return_X_y=True)

In [2]: def gaussian(dist, band):
   ...:     u = dist / band
   ...:     return np.exp(-((u / 2) ** 2)) / (np.sqrt(2 * np.pi))
   ...: 
   ...: def gaussian_decay(dist, band):
   ...:     u = dist / band
   ...:     return np.exp(-((u / 2) ** 2))
   ...: 

In [3]: dist = np.linspace(0, 100, X.shape[0])
   ...: 

In [4]: weights = gaussian(dist, 25)
   ...: weights
Out[4]: 
array([0.39894228, 0.39893408, 0.39890946, 0.39886844, 0.39881102,
       0.3987372 , 0.398647  , 0.39854042, 0.39841749, 0.39827821,
       0.3981226 , 0.39795068, 0.39776247, 0.397558  , 0.39733728,
       0.39710036, 0.39684725, 0.39657799, 0.39629261, 0.39599115,
       0.39567364, 0.39534012, 0.39499063, 0.39462522, 0.39424393,
       0.39384681, 0.3934339 , 0.39300526, 0.39256094, 0.39210099,
       0.39162547, 0.39113443, 0.39062795, 0.39010607, 0.38956886,
       0.38901639, 0.38844873, 0.38786594, 0.38726809, 0.38665526,
       0.38602751, 0.38538494, 0.38472761, 0.3840556 , 0.38336899,
       0.38266787, 0.38195232, 0.38122243, 0.38047828, 0.37971996,
       0.37894756, 0.37816118, 0.37736091, 0.37654685, 0.37571908,
       0.37487771, 0.37402284, 0.37315457, 0.372273  , 0.37137824,
       0.37047039, 0.36954956, 0.36861585, 0.36766937, 0.36671024,
       0.36573857, 0.36475447, 0.36375805, 0.36274944, 0.36172874,
       0.36069607, 0.35965156, 0.35859532, 0.35752747, 0.35644815,
       0.35535746, 0.35425554, 0.3531425 , 0.35201849, 0.35088362,
       0.34973802, 0.34858182, 0.34741515, 0.34623814, 0.34505093,
       0.34385364, 0.34264642, 0.34142938, 0.34020268, 0.33896643,
       0.33772079, 0.33646589, 0.33520185, 0.33392884, 0.33264697,
       0.33135639, 0.33005724, 0.32874966, 0.32743379, 0.32610978,
       0.32477776, 0.32343787, 0.32209026, 0.32073508, 0.31937246,
       0.31800254, 0.31662548, 0.31524141, 0.31385048, 0.31245284,
       0.31104862, 0.30963798, 0.30822106, 0.306798  , 0.30536895,
       0.30393405, 0.30249345, 0.3010473 , 0.29959573, 0.29813891,
       0.29667696, 0.29521003, 0.29373828, 0.29226184, 0.29078086,
       0.28929548, 0.28780585, 0.28631212, 0.28481442, 0.2833129 ,
       0.28180771, 0.28029898, 0.27878686, 0.27727149, 0.27575302,
       0.27423158, 0.27270732, 0.27118037, 0.26965088, 0.26811899,
       0.26658484, 0.26504856, 0.26351029, 0.26197018, 0.26042835,
       0.25888495, 0.25734011, 0.25579397, 0.25424665, 0.2526983 ,
       0.25114905, 0.24959903, 0.24804838, 0.24649721, 0.24494567,
       0.24339389, 0.24184199, 0.24029009, 0.23873834, 0.23718685,
       0.23563575, 0.23408516, 0.23253521, 0.23098602, 0.22943772,
       0.22789041, 0.22634424, 0.2247993 , 0.22325573, 0.22171363,
       0.22017313, 0.21863434, 0.21709738, 0.21556235, 0.21402937,
       0.21249855, 0.21097   , 0.20944383, 0.20792015, 0.20639906,
       0.20488068, 0.20336509, 0.20185242, 0.20034275, 0.1988362 ,
       0.19733286, 0.19583283, 0.19433621, 0.19284309, 0.19135357,
       0.18986775, 0.18838571, 0.18690756, 0.18543337, 0.18396325,
       0.18249727, 0.18103553, 0.1795781 , 0.17812509, 0.17667656,
       0.17523261, 0.1737933 , 0.17235873, 0.17092897, 0.1695041 ,
       0.16808419, 0.16666931, 0.16525955, 0.16385498, 0.16245566,
       0.16106166, 0.15967306, 0.15828991, 0.1569123 , 0.15554027,
       0.1541739 , 0.15281325, 0.15145837, 0.15010934, 0.14876619,
       0.14742901, 0.14609783, 0.14477271, 0.14345372, 0.14214089,
       0.14083429, 0.13953395, 0.13823994, 0.13695229, 0.13567105,
       0.13439628, 0.133128  , 0.13186627, 0.13061112, 0.1293626 ,
       0.12812075, 0.12688559, 0.12565717, 0.12443553, 0.1232207 ,
       0.12201271, 0.12081159, 0.11961737, 0.11843008, 0.11724976,
       0.11607643, 0.11491011, 0.11375083, 0.11259862, 0.11145349,
       0.11031547, 0.10918457, 0.10806083, 0.10694425, 0.10583486,
       0.10473266, 0.10363768, 0.10254994, 0.10146943, 0.10039618,
       0.09933019, 0.09827148, 0.09722006, 0.09617592, 0.09513909,
       0.09410957, 0.09308735, 0.09207246, 0.09106488, 0.09006462,
       0.08907168, 0.08808607, 0.08710778, 0.08613682, 0.08517317,
       0.08421684, 0.08326782, 0.08232611, 0.0813917 , 0.08046459,
       0.07954476, 0.07863222, 0.07772694, 0.07682893, 0.07593817,
       0.07505465, 0.07417836, 0.07330928, 0.07244741, 0.07159272,
       0.07074521, 0.06990485, 0.06907163, 0.06824554, 0.06742656,
       0.06661466, 0.06580983, 0.06501205, 0.0642213 , 0.06343756,
       0.06266081, 0.06189102, 0.06112817, 0.06037224, 0.05962321,
       0.05888105, 0.05814573, 0.05741724, 0.05669554, 0.05598061,
       0.05527242, 0.05457094, 0.05387616, 0.05318802, 0.05250652,
       0.05183162, 0.05116329, 0.0505015 , 0.04984622, 0.04919742,
       0.04855506, 0.04791912, 0.04728957, 0.04666636, 0.04604948,
       0.04543888, 0.04483453, 0.0442364 , 0.04364445, 0.04305866,
       0.04247898, 0.04190537, 0.04133782, 0.04077627, 0.0402207 ,
       0.03967107, 0.03912733, 0.03858947, 0.03805743, 0.03753118,
       0.03701068, 0.03649591, 0.03598681, 0.03548335, 0.0349855 ,
       0.03449322, 0.03400646, 0.03352519, 0.03304938, 0.03257898,
       0.03211395, 0.03165425, 0.03119986, 0.03075072, 0.0303068 ,
       0.02986806, 0.02943446, 0.02900597, 0.02858253, 0.02816412,
       0.02775069, 0.02734221, 0.02693863, 0.02653991, 0.02614602,
       0.02575692, 0.02537256, 0.02499292, 0.02461794, 0.02424758,
       0.02388182, 0.02352061, 0.02316391, 0.02281167, 0.02246387,
       0.02212047, 0.02178142, 0.02144668, 0.02111622, 0.02078999,
       0.02046796, 0.02015009, 0.01983634, 0.01952668, 0.01922106,
       0.01891944, 0.01862179, 0.01832807, 0.01803824, 0.01775226,
       0.0174701 , 0.01719171, 0.01691707, 0.01664612, 0.01637885,
       0.0161152 , 0.01585514, 0.01559864, 0.01534566, 0.01509616,
       0.0148501 , 0.01460746, 0.01436819, 0.01413225, 0.01389962,
       0.01367026, 0.01344412, 0.01322119, 0.01300141, 0.01278477,
       0.01257121, 0.01236072, 0.01215325, 0.01194877, 0.01174725,
       0.01154865, 0.01135294, 0.01116009, 0.01097006, 0.01078283,
       0.01059836, 0.01041661, 0.01023756, 0.01006117, 0.00988742,
       0.00971626, 0.00954768, 0.00938163, 0.0092181 , 0.00905704,
       0.00889843, 0.00874224, 0.00858843, 0.00843699, 0.00828787,
       0.00814106, 0.00799651, 0.00785421, 0.00771413, 0.00757623,
       0.00744049, 0.00730688])

In [5]: reg = LinearRegression().fit(X, y, sample_weight=weights)

In [6]: reg.coef_
   ...: 
Out[6]: 
array([ -37.07281815, -279.89817078,  516.04632924,  278.483909  ,
       -583.14095776,  208.28112115,   76.73856879,  224.27818248,
        722.02267047,   98.53405821])

In [7]: weights_decay = gaussian_decay(dist, 25)
   ...: weights_decay
Out[7]: 
array([1.        , 0.99997943, 0.99991773, 0.99981491, 0.99967097,
       0.99948594, 0.99925984, 0.9989927 , 0.99868454, 0.99833541,
       0.99794536, 0.99751442, 0.99704265, 0.99653012, 0.99597687,
       0.99538299, 0.99474854, 0.99407361, 0.99335826, 0.99260261,
       0.99180672, 0.99097072, 0.99009468, 0.98917874, 0.98822298,
       0.98722754, 0.98619254, 0.98511809, 0.98400434, 0.98285142,
       0.98165947, 0.98042863, 0.97915905, 0.9778509 , 0.97650432,
       0.97511949, 0.97369656, 0.97223572, 0.97073714, 0.969201  ,
       0.96762748, 0.96601678, 0.96436909, 0.96268462, 0.96096355,
       0.9592061 , 0.95741249, 0.95558291, 0.95371761, 0.95181678,
       0.94988068, 0.94790952, 0.94590353, 0.94386297, 0.94178807,
       0.93967907, 0.93753623, 0.9353598 , 0.93315004, 0.9309072 ,
       0.92863155, 0.92632337, 0.92398291, 0.92161045, 0.91920627,
       0.91677065, 0.91430387, 0.91180622, 0.909278  , 0.90671948,
       0.90413097, 0.90151276, 0.89886516, 0.89618847, 0.893483  ,
       0.89074905, 0.88798695, 0.88519699, 0.8823795 , 0.87953479,
       0.8766632 , 0.87376504, 0.87084064, 0.86789032, 0.86491442,
       0.86191326, 0.85888719, 0.85583654, 0.85276165, 0.84966285,
       0.84654049, 0.8433949 , 0.84022645, 0.83703546, 0.83382229,
       0.83058729, 0.82733081, 0.8240532 , 0.82075481, 0.81743599,
       0.81409711, 0.81073852, 0.80736057, 0.80396362, 0.80054803,
       0.79711416, 0.79366238, 0.79019303, 0.7867065 , 0.78320312,
       0.77968328, 0.77614732, 0.77259562, 0.76902854, 0.76544644,
       0.76184969, 0.75823864, 0.75461367, 0.75097514, 0.74732341,
       0.74365885, 0.73998181, 0.73629267, 0.73259178, 0.72887952,
       0.72515623, 0.72142229, 0.71767805, 0.71392388, 0.71016013,
       0.70638717, 0.70260535, 0.69881503, 0.69501656, 0.69121031,
       0.68739663, 0.68357587, 0.67974839, 0.67591453, 0.67207465,
       0.66822909, 0.66437821, 0.66052235, 0.65666186, 0.65279707,
       0.64892834, 0.645056  , 0.64118039, 0.63730185, 0.63342071,
       0.62953732, 0.625652  , 0.62176508, 0.61787689, 0.61398775,
       0.610098  , 0.60620796, 0.60231794, 0.59842827, 0.59453926,
       0.59065122, 0.58676448, 0.58287933, 0.57899609, 0.57511507,
       0.57123656, 0.56736086, 0.56348829, 0.55961912, 0.55575366,
       0.5518922 , 0.54803503, 0.54418242, 0.54033468, 0.53649207,
       0.53265487, 0.52882337, 0.52499783, 0.52117853, 0.51736573,
       0.5135597 , 0.50976069, 0.50596898, 0.50218481, 0.49840844,
       0.49464012, 0.4908801 , 0.48712863, 0.48338594, 0.47965227,
       0.47592786, 0.47221295, 0.46850777, 0.46481253, 0.46112747,
       0.45745281, 0.45378877, 0.45013555, 0.44649338, 0.44286247,
       0.43924301, 0.43563521, 0.43203927, 0.42845539, 0.42488376,
       0.42132457, 0.41777801, 0.41424427, 0.41072352, 0.40721594,
       0.40372171, 0.400241  , 0.39677397, 0.3933208 , 0.38988164,
       0.38645666, 0.38304601, 0.37964984, 0.37626831, 0.37290155,
       0.36954972, 0.36621295, 0.36289138, 0.35958515, 0.35629438,
       0.3530192 , 0.34975975, 0.34651614, 0.34328848, 0.3400769 ,
       0.33688151, 0.33370241, 0.33053972, 0.32739353, 0.32426396,
       0.32115108, 0.31805501, 0.31497583, 0.31191362, 0.30886849,
       0.3058405 , 0.30282974, 0.29983628, 0.2968602 , 0.29390157,
       0.29096046, 0.28803693, 0.28513105, 0.28224287, 0.27937246,
       0.27651986, 0.27368514, 0.27086833, 0.26806948, 0.26528864,
       0.26252585, 0.25978115, 0.25705457, 0.25434614, 0.2516559 ,
       0.24898387, 0.24633007, 0.24369454, 0.24107729, 0.23847834,
       0.2358977 , 0.23333539, 0.23079142, 0.22826579, 0.22575852,
       0.2232696 , 0.22079904, 0.21834683, 0.21591298, 0.21349747,
       0.21110031, 0.20872147, 0.20636095, 0.20401873, 0.20169481,
       0.19938915, 0.19710174, 0.19483255, 0.19258157, 0.19034877,
       0.18813411, 0.18593757, 0.18375912, 0.18159872, 0.17945634,
       0.17733194, 0.17522547, 0.17313691, 0.17106621, 0.16901332,
       0.16697819, 0.16496078, 0.16296105, 0.16097893, 0.15901439,
       0.15706735, 0.15513778, 0.1532256 , 0.15133077, 0.14945323,
       0.1475929 , 0.14574974, 0.14392368, 0.14211464, 0.14032258,
       0.13854741, 0.13678907, 0.1350475 , 0.13332261, 0.13161434,
       0.12992261, 0.12824735, 0.12658849, 0.12494594, 0.12331963,
       0.12170949, 0.12011542, 0.11853736, 0.11697522, 0.11542892,
       0.11389837, 0.11238349, 0.1108842 , 0.10940042, 0.10793204,
       0.106479  , 0.1050412 , 0.10361855, 0.10221096, 0.10081835,
       0.09944062, 0.09807768, 0.09672944, 0.09539582, 0.09407671,
       0.09277202, 0.09148167, 0.09020555, 0.08894358, 0.08769565,
       0.08646167, 0.08524155, 0.0840352 , 0.0828425 , 0.08166338,
       0.08049773, 0.07934545, 0.07820645, 0.07708063, 0.07596789,
       0.07486813, 0.07378126, 0.07270718, 0.07164578, 0.07059698,
       0.06956067, 0.06853675, 0.06752512, 0.06652569, 0.06553836,
       0.06456302, 0.06359959, 0.06264795, 0.06170801, 0.06077968,
       0.05986285, 0.05895742, 0.0580633 , 0.05718039, 0.05630858,
       0.05544779, 0.05459791, 0.05375885, 0.0529305 , 0.05211278,
       0.05130557, 0.05050879, 0.04972234, 0.04894612, 0.04818004,
       0.047424  , 0.0466779 , 0.04594165, 0.04521515, 0.04449831,
       0.04379104, 0.04309323, 0.0424048 , 0.04172564, 0.04105568,
       0.04039482, 0.03974295, 0.0391    , 0.03846586, 0.03784046,
       0.03722369, 0.03661547, 0.0360157 , 0.0354243 , 0.03484118,
       0.03426625, 0.03369942, 0.0331406 , 0.03258971, 0.03204666,
       0.03151136, 0.03098373, 0.03046368, 0.02995112, 0.02944598,
       0.02894817, 0.0284576 , 0.0279742 , 0.02749787, 0.02702855,
       0.02656614, 0.02611057, 0.02566175, 0.02521961, 0.02478407,
       0.02435505, 0.02393248, 0.02351626, 0.02310634, 0.02270263,
       0.02230505, 0.02191353, 0.021528  , 0.02114839, 0.02077461,
       0.0204066 , 0.02004428, 0.01968759, 0.01933645, 0.01899079,
       0.01865054, 0.01831564])

In [8]: reg = LinearRegression().fit(X, y, sample_weight=weights_decay)

In [9]: reg.coef_
   ...: 
Out[9]: 
array([ -37.07281815, -279.89817078,  516.04632924,  278.483909  ,
       -583.14095776,  208.28112115,   76.73856879,  224.27818248,
        722.02267047,   98.53405821])

[ljwolf]

Yep! As noted above,

the constants should cancel out within the WLS estimator, no? However, we would get in trouble for many other estimators or statistics, so the kernel should be fully specified.

Any OLS weighted estimator will divide by (and multiply by) the constant, cancelling it out. So, you just need a kernel proportional to the true kernel---hence a decay function works.

But, this isn't the case for all estimators.

A really simple case is the Ridge regression:

from sklearn.linear_model import Ridge
lm.Ridge().fit(X, y, sample_weight=weights).coef_
# array([ 20.39085409, -11.79013363, 105.86326622,  72.82236978,
#        17.65506374,   5.24382731, -65.99424209,  63.22310885,
#       107.32876914,  63.26914272])
lm.Ridge().fit(X, y sample_weight=weights_decay).coef_
# array([  22.46055986,  -46.89069324,  192.11483436,  125.35432132,
#         12.8104851 ,  -16.04140752, -107.75469729,   93.83849542,
#        193.11117961,   98.29279074])

Only the proper kernel here is correct.

Hence, mgwr needs to implement proper kernels, because this will cause random errors for models in the wild that people will seek to implement in the future.

[ljwolf]

See also statsmodels, which uses proper kernels for gaussian and tricube, while sklearn uses Normalized to refer to K(0)=1..

[martinfleis]

Only the proper kernel here is correct.

Do you have any pointers where could I read more on this? I understand they the results are different (and why) but don't understand what makes one correct and the other wrong in this context...

[ljwolf]

Hmm... this is proving quite challenging due to how heterogeneous the term "kernel" is.

The original Cleveland and Devlin work on local regression just uses "Weighting function". They omit the constants. Kernel density estimation lit retains the constants. Gaussian process literature (popular in ML) uses "kernel" to refer colloquially to a covariance function (the decay sense), which tend to satisfy k(xi,xj)=1, but are not pdfs. Both literatures are valid, but it's important for us to be clear.

Hence, the ridge regression estimate is "correct" in the sense that that it is the true ridge regression estimate for a Gaussian kernel, while the other is an estimate for a different (RBF, squared exponential) improper kernel. The Gaussian kernel is a proper RBF kernel.

[DOSull]

My thought would be to offer a keyword argument either boolean or some string constant depending on how many options you want to support. One of the options should certainly be "volume_standardised" and another might be "zero_standardised" or whatever language seems most appropriate. For a kernel function $k(r)$, the former would scale so that

$$\int_{r=0}^{\infty}\int_{\theta=0}^{2\pi}r\mathop{d\theta}\mathop{dr}=1$$

[something like that, I might have that integral garbled], and the latter would scale so that $k(0)=1$. A third option might be "raw" where you don't apply any constant and just use the functional form of the kernel 'bare'.

I'm not sure what the default should be. I am certain that it really does matter when people are using a kernel to do things like population smoothing. It seems not to matter at all if all that is required is a 'decay' function, although consistency is likely desirable in case it affects comparisons across different functional forms of kernel function if someone is exploring the effects of such choices.

Keeping a new keyword option consistent with previous behaviour might require adding an additional "legacy" option, and having it be the default so that code continues to behave as it has in the past.