Return mesh size vector vector corresponding to best decision vector found

[sblelong]

Currently, result = solve(problem, x0) yields a tuple with only the best feasible and infeasible found points + their related objective values.

It may be of use to return other outputs yielded by NOMAD via this object, e.g. the actual number of blackbox evaluations performed and the last value of the mesh size parameter.

[salomonl]

@sblelong There are two reasons why I am reluctant to do that.

• Changing the C interface on which this project depends; In order to obtain the last mesh size vector values, we would need to modify the C interface. This is not a problem, but if other parameters and statistics are requested by other users, they would need to be added, which would complexify the C interface. I would prefer to keep it simple. There are some techniques that can be used to bypass this. For the mesh size vector, you can experiment with the display_stats options (see MESH_INDEX), and at the end of the execution, read the logs to collect the values. Otherwise, you need to use the C++ API. To get the actual number of blackbox evaluations, do this:

nbb_evals = 0
function bb(x)
  # Define your blackbox
  f = x[1]^2 + x[2]^2
  c = 1 - x[1]
  # Set the flags
  success = true
  count_eval = true
  bb_outputs = [f; c]
  # Update the current number of blackbox evaluations.
  if count_eval
      nbb_evals += 1
  end
  return (success, count_eval, bb_outputs)
end

You can put that inside a function to limit the scope of nbb_evals. Using this technique, you can also define other statistics, such as average evaluation time.

• Solving a linear equality problem involves solving a smaller problem in terms of the number of variables. After optimization, a transformation is applied to the low-dimensional solution to obtain the full-dimensional solution. While I can obtain the mesh size vector for the low-dimensional problem, I am not aware of any transformation that can produce the full-dimensional mesh size vector from this low-dimensional mesh size vector. I also do not want to return different result attributes for the linear equality case and the general case.

[sblelong]

@salomonl

• This might be a naive question, as I have very poor (if any) knowledge of how NOMAD's native API is interfaced with Julia here. But: why not just retrieve all the information that the C++ API can provide, instead of waiting for future user's requests? That might encourage the use of this interface over the C++ version.
• Your suggested fix for counting the actual number of evaluations will be useful to me, thanks. It's silly of me, not to have thought of it before.
• About solving problems with linear equalities. I'd find it reasonable to return the latest mesh size used in the lower-dimensional problem, with the exact same structure as for other problems. That would require thorough documentation though, including the notations from Mathilde Peyrega's work (by that I mean $n_x$, $n_z$ and $\varphi$). A quick warning in the documentation about that would do the trick, in my opinion.

[salomonl]

@sblelong

• This is not a naive question. Initially, the Julia interface was implemented using the Cxx.jl package. Unfortunately, we encountered many bugs due to the lack of robustness of this package at that time. You can also see on their repertory that it only supports older versions of Julia. After discussion with @amontoison and others who were familiar with interfacing C/C++ software with Julia, we concluded that the simplest solution was to provide a C api to Nomad and call it directly from Julia. Also, the reason we limit ourselves to a subset of the functionalities of Nomad is the management of exceptions in C++, which is difficult to do with Julia. By choosing a subset of functionalities, we can avoid a segmentation fault by catching errors before calling the C API, and limit the number of bugs coming from Nomad that an user could trigger.
• I do not like it because of the potential incompatibility between the dimensions of the optimal solution and the mesh size vector. For the lower-dimensional subproblem, there is a direct correspondence between the mesh size vector and the optimal solution to the problem. It does not make sense to return the optimal lower-dimensional solution because the user wants the full-dimensional solution, not the lower-dimensional one. It is not up to them to call the $phi$ transformation; it should be done automatically.

[sblelong]

@salomonl

• Alright then I'll leave it to you to decide when/whether that could be doable. Thanks for the explanation!
• I get your point, but I think that should be on the user to at least read about how linear equalities are handled here, and why the $\delta$ they'd get in that situation wouldn't have the same dimension as their solution. Even more so as what is displayed while NOMAD is running are the tentative points generated while solving the lower-level problem, so a user should've noticed by then that something is happening behind the curtain. I'm not saying they should have to apply $\varphi$ at any moment, that is indeed not up to them, I agree. Just the value of $\delta$ in the lower-dimensional problem could still give some insight as to what happened during the optimization.

[kockahonza]

Hi, joining in on this as I have a similar issue to deal with. In particular I want to retrieve/store the history of all evaluations of the blackbox that NOMAD made. As far as I see the only way right now would be use DISPLAY_STATS and somehow retrieve it from stdout? I have no idea how to do that without somehow piping it to a file which itself I don't know how to do from julia itself.

However, I noticed that NOMAD itself has a STATS_FILE option which allows to sidestep at least a part of the issue. Would it be possible to at least expose that? That seems like it should be relatively easy and would provide a way to save NOMADs output without having to fiddle with stdout somehow.

[salomonl]

@kockahonza You can use the same technique as mentioned below:

history_cache = Vector{Float64}[]
function bb(x)
  # Define your blackbox
  ...
  # Set the flags
  success = true
  count_eval = true
  bb_outputs = [f; c]
  # Update the history cache
  append!(history_cache, [bb_outputs])
  return (success, count_eval, bb_outputs)
end

As a bonus, no need to in this case to deal with io issues.
I will change the title of this issue. If you think it is pertinent, please open an issue for each functionnality you want to have.

[salomonl]

@amontoison Could you change the title of this issue to "Return mesh size vector vector corresponding to best decision vector found", please ?

[amontoison]

Done @salomonl 👍

[kockahonza]

@kockahonza You can use the same technique as mentioned below:

history_cache = Vector{Float64}[]
function bb(x)

Define your blackbox

...

Set the flags

success = true
count_eval = true
bb_outputs = [f; c]

Update the history cache

append!(history_cache, [bb_outputs])
return (success, count_eval, bb_outputs)
end

As a bonus, no need to in this case to deal with io issues. I will change the title of this issue. If you think it is pertinent, please open an issue for each functionnality you want to have.

Aaah, yes, thank you for that tip, you saved me having to deal with regex for extracting the numbers from the output!

[amontoison]

@salomonl what do you think of adding a nomad_set▪︎ and nomad_get in the C interface and it passes a Ptr{Cvoid} from Julia to get any result or set any option?
We open a can of worms but we can still check what is passed from Julia and it is robust to future modifications of NOMAD.

[salomonl]

@amontoison The core of the problem is how to deal with exceptions coming from the C++ Nomad library.

• Some functions in the C API allow any string, array, float or bool parameter to be passed to Nomad. In any case, we will need to control what the user can pass with these functions. The initial reason for choosing a 'NomadOptions' approach was to prevent users from incorrectly using the C wrapper functions and ensure that only the correct functions are called when everything is "valid". We could also use a dictionary of keyword parameters with their arguments taken directly from the Nomad website. However, we would still need to check the user's inputs and update the keywords, as the parameter names are prone to change with each Nomad release.
• Obtaining the different optimal statistical values of Nomad is not straightforward and, in this case, requires a change to the C API. The new version has been updated (we should now be able to solve multiobjective problems and handle a certain form of parallelism), but there is a compiling issue on a certain platform (I do not manage to debug this due to cryptic linking errors) that prevents its deployment on julia.

[amontoison]

I can have a look for the cross-compilation issue.